[jira] [Created] (HUDI-1728) Addressing ClassNotFound when using HiveMetastoreBasedLockProvider
Nishith Agarwal created HUDI-1728: - Summary: Addressing ClassNotFound when using HiveMetastoreBasedLockProvider Key: HUDI-1728 URL: https://issues.apache.org/jira/browse/HUDI-1728 Project: Apache Hudi Issue Type: Bug Components: Common Core Reporter: Nishith Agarwal Assignee: Nishith Agarwal -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-io commented on pull request #2730: [DO NOT MERGE]Verify IT spark3
codecov-io commented on pull request #2730: URL: https://github.com/apache/hudi/pull/2730#issuecomment-808660342 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2730?src=pr=h1) Report > Merging [#2730](https://codecov.io/gh/apache/hudi/pull/2730?src=pr=desc) (cae623b) into [release-0.8.0](https://codecov.io/gh/apache/hudi/commit/399604412b9a5f59ae88a0dac98d94e61638a43a?el=desc) (3996044) will **decrease** coverage by `42.33%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2730?src=pr=tree) ```diff @@ Coverage Diff @@ ## release-0.8.0 #2730 +/- ## === - Coverage51.73% 9.40% -42.34% + Complexity3601 48 -3553 === Files 476 54 -422 Lines225951989-20606 Branches 2409 236 -2173 === - Hits 11689 187-11502 + Misses98881789 -8099 + Partials 1018 13 -1005 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.40% <ø> (-60.34%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2730?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2730/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2730/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2730/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2730/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2730/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2730/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2730/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2730/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | | | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2730/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | | |
[GitHub] [hudi] codecov-io commented on pull request #2729: [DO NOT MERGE] Verify IT scala2.12
codecov-io commented on pull request #2729: URL: https://github.com/apache/hudi/pull/2729#issuecomment-808659555 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2729?src=pr=h1) Report > Merging [#2729](https://codecov.io/gh/apache/hudi/pull/2729?src=pr=desc) (e8805b8) into [release-0.8.0](https://codecov.io/gh/apache/hudi/commit/399604412b9a5f59ae88a0dac98d94e61638a43a?el=desc) (3996044) will **decrease** coverage by `42.33%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2729?src=pr=tree) ```diff @@ Coverage Diff @@ ## release-0.8.0 #2729 +/- ## === - Coverage51.73% 9.40% -42.34% + Complexity3601 48 -3553 === Files 476 54 -422 Lines225951989-20606 Branches 2409 236 -2173 === - Hits 11689 187-11502 + Misses98881789 -8099 + Partials 1018 13 -1005 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.40% <ø> (-60.34%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2729?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2729/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2729/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2729/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2729/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2729/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2729/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2729/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2729/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | | | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2729/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | | |
[GitHub] [hudi] garyli1019 opened a new pull request #2730: [DO NOT MERGE]Verify IT spark3
garyli1019 opened a new pull request #2730: URL: https://github.com/apache/hudi/pull/2730 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] garyli1019 opened a new pull request #2729: [DO NOT MERGE] Verify IT scala2.12
garyli1019 opened a new pull request #2729: URL: https://github.com/apache/hudi/pull/2729 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch release-0.8.0 updated: [HOTFIX] close spark session in functional test suite and disable spark3 test for spark2 (#2727)
This is an automated email from the ASF dual-hosted git repository.
garyli pushed a commit to branch release-0.8.0
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/release-0.8.0 by this push:
new 3996044 [HOTFIX] close spark session in functional test suite and
disable spark3 test for spark2 (#2727)
3996044 is described below
commit 399604412b9a5f59ae88a0dac98d94e61638a43a
Author: Gary Li
AuthorDate: Fri Mar 26 20:58:29 2021 -0700
[HOTFIX] close spark session in functional test suite and disable spark3
test for spark2 (#2727)
---
.../org/apache/hudi/testutils/FunctionalTestHarness.java | 16
hudi-spark-datasource/hudi-spark2/pom.xml| 2 +-
hudi-spark-datasource/hudi-spark3/pom.xml| 7 +++
.../hudi/utilities/testutils/UtilitiesTestBase.java | 9 +
pom.xml | 2 ++
5 files changed, 35 insertions(+), 1 deletion(-)
diff --git
a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/FunctionalTestHarness.java
b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/FunctionalTestHarness.java
index fc02e6d..e391abf 100644
---
a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/FunctionalTestHarness.java
+++
b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/FunctionalTestHarness.java
@@ -152,6 +152,8 @@ public class FunctionalTestHarness implements
SparkProvider, DFSProvider, Hoodie
hdfsTestService.stop();
hdfsTestService = null;
+jsc.close();
+jsc = null;
spark.stop();
spark = null;
}));
@@ -166,5 +168,19 @@ public class FunctionalTestHarness implements
SparkProvider, DFSProvider, Hoodie
for (FileStatus f : fileStatuses) {
fs.delete(f.getPath(), true);
}
+if (hdfsTestService != null) {
+ hdfsTestService.stop();
+ hdfsTestService = null;
+}
+if (spark != null) {
+ spark.stop();
+ spark = null;
+}
+if (jsc != null) {
+ jsc.close();
+ jsc = null;
+}
+sqlContext = null;
+context = null;
}
}
diff --git a/hudi-spark-datasource/hudi-spark2/pom.xml
b/hudi-spark-datasource/hudi-spark2/pom.xml
index 91718ab..91f4849 100644
--- a/hudi-spark-datasource/hudi-spark2/pom.xml
+++ b/hudi-spark-datasource/hudi-spark2/pom.xml
@@ -151,7 +151,7 @@
org.scala-lang
scala-library
- ${scala11.version}
+ ${scala.version}
diff --git a/hudi-spark-datasource/hudi-spark3/pom.xml
b/hudi-spark-datasource/hudi-spark3/pom.xml
index d6028bb..813c4e7 100644
--- a/hudi-spark-datasource/hudi-spark3/pom.xml
+++ b/hudi-spark-datasource/hudi-spark3/pom.xml
@@ -125,6 +125,13 @@
+org.apache.maven.plugins
+maven-surefire-plugin
+
+ ${skip.hudi-spark3.unit.tests}
+
+
+
org.apache.rat
apache-rat-plugin
diff --git
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/UtilitiesTestBase.java
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/UtilitiesTestBase.java
index 6efd468..0adef52 100644
---
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/UtilitiesTestBase.java
+++
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/UtilitiesTestBase.java
@@ -124,15 +124,19 @@ public class UtilitiesTestBase {
public static void cleanupClass() {
if (hdfsTestService != null) {
hdfsTestService.stop();
+ hdfsTestService = null;
}
if (hiveServer != null) {
hiveServer.stop();
+ hiveServer = null;
}
if (hiveTestService != null) {
hiveTestService.stop();
+ hiveTestService = null;
}
if (zookeeperTestService != null) {
zookeeperTestService.stop();
+ zookeeperTestService = null;
}
}
@@ -150,6 +154,11 @@ public class UtilitiesTestBase {
TestDataSource.resetDataGen();
if (jsc != null) {
jsc.stop();
+ jsc = null;
+}
+if (sparkSession != null) {
+ sparkSession.close();
+ sparkSession = null;
}
if (context != null) {
context = null;
diff --git a/pom.xml b/pom.xml
index 8ee6831..db1a798 100644
--- a/pom.xml
+++ b/pom.xml
@@ -133,6 +133,7 @@
${skipTests}
${skipTests}
${skipTests}
+${skipTests}
UTF-8
${project.basedir}
provided
@@ -1424,6 +1425,7 @@
${scala12.version}
2.12
+true
[GitHub] [hudi] garyli1019 merged pull request #2727: [HOTFIX] fix spark session leak in FunctionalTestHarness
garyli1019 merged pull request #2727: URL: https://github.com/apache/hudi/pull/2727 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] Tandoy opened a new issue #2728: [SUPPORT]Hive sync error by using run_sync_tool.sh
Tandoy opened a new issue #2728: URL: https://github.com/apache/hudi/issues/2728 **Steps to reproduce the behavior:** `./run_sync_tool_sh --jdbc-url jdbc:hive://dxbigdata102:1000 \ --user appuser \ --pass '' \ --base-path 'hdfs://dxbigdata101:8020/user/hudi/test/data/hudi_trips_cow' \ --database test \ --table hudi_trips_cow ` **Expected behavior:** to sync a hoodie HDFS table with a hive metastore table **Environment Description:** Hudi version : 0.7.0 Spark version : 2.4.0.cloudera2 Hadoop version : 2.6.0-cdh5.13.3 Hive version : 1.1.0-cdh5.13.3 Storage (HDFS/S3/GCS..) : HDFS Running on Docker? (yes/no) :no **Stacktrace:** Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/LogManager at org.apache.hudi.hive.HiveSyncTool.(HiveSyncTool.java:55) Caused by: java.lang.ClassNotFoundException: org.apache.log4j.LogManager at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 1 more -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] umehrot2 edited a comment on pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
umehrot2 edited a comment on pull request #2651: URL: https://github.com/apache/hudi/pull/2651#issuecomment-808606053 @pengzhiwei2018 I was testing Hudi without this patch via Spark SQL and I am a little confused. With Spark SQL I see partition pruning already works seamlessly for Hudi. Just start spark sql with: ``` spark-sql --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.hadoop.mapreduce.input.pathFilter.class=org.apache.hudi.hadoop.HoodieROTablePathFilter" --jars /usr/lib/hudi/hudi-spark-bundle.jar ``` Spark is able to get the partition schema from the catalog using `CatalogFileIndex` and do the partition pruning. So this partition pruning support we are adding, is this to be able to support partition pruning for datasource based queries ? I think for hive style partition tables pruning should have already worked via Spark datasource too, because Spark tries to identify partition columns from the path, but not sure why it does not work. I want to understand clearly what we are gaining when this PR gets merged, is it partition pruning for Spark datasource queries ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] umehrot2 commented on pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
umehrot2 commented on pull request #2651: URL: https://github.com/apache/hudi/pull/2651#issuecomment-808606053 @pengzhiwei2018 I was testing Hudi without this patch via Spark SQL and I am a little confused. With Spark SQL I see partition pruning already works seamlessly for Hudi. Just start spark sql with: ``` spark-sql --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.hadoop.mapreduce.input.pathFilter.class=org.apache.hudi.hadoop.HoodieROTablePathFilter" --jars /usr/lib/hudi/hudi-spark-bundle.jar ``` Spark is able to get the partition schema from the catalog using `CatalogFileIndex` and do the partition pruning. So this partition pruning support we are adding, is this to be able to support partition pruning for datasource based queries ? I think for hive style partition tables pruning should have already worked via Spark datasource too, because Spark tries to identify partition columns from the path, but not sure why it does not work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] satishkotha commented on pull request #2532: [HUDI-1534]HiveSyncTool-It is not necessary to use JDBC and MetaStoreClient at the same time
satishkotha commented on pull request #2532: URL: https://github.com/apache/hudi/pull/2532#issuecomment-808605301 @lhjzmn we discussed this in one of the OSS meetings. we want to keep jdbc support for backward compatibility reasons. we can separate out MetastoreClient as a separate class though. please let me know what you think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1727) Fix default value for cleaner commits retained in configurations page
[ https://issues.apache.org/jira/browse/HUDI-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1727: -- Labels: docs user-support-issues (was: docs) > Fix default value for cleaner commits retained in configurations page > - > > Key: HUDI-1727 > URL: https://issues.apache.org/jira/browse/HUDI-1727 > Project: Apache Hudi > Issue Type: Bug > Components: Docs >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: docs, user-support-issues > > [http://hudi.apache.org/docs/configurations.html#retainCommits] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1727) Fix default value for cleaner commits retained in configurations page
[ https://issues.apache.org/jira/browse/HUDI-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1727: -- Labels: docs (was: ) > Fix default value for cleaner commits retained in configurations page > - > > Key: HUDI-1727 > URL: https://issues.apache.org/jira/browse/HUDI-1727 > Project: Apache Hudi > Issue Type: Bug > Components: Docs >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: docs > > [http://hudi.apache.org/docs/configurations.html#retainCommits] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1727) Fix default value for cleaner commits retained in configurations page
sivabalan narayanan created HUDI-1727: - Summary: Fix default value for cleaner commits retained in configurations page Key: HUDI-1727 URL: https://issues.apache.org/jira/browse/HUDI-1727 Project: Apache Hudi Issue Type: Bug Components: Docs Reporter: sivabalan narayanan [http://hudi.apache.org/docs/configurations.html#retainCommits] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1727) Fix default value for cleaner commits retained in configurations page
[ https://issues.apache.org/jira/browse/HUDI-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-1727: - Assignee: sivabalan narayanan > Fix default value for cleaner commits retained in configurations page > - > > Key: HUDI-1727 > URL: https://issues.apache.org/jira/browse/HUDI-1727 > Project: Apache Hudi > Issue Type: Bug > Components: Docs >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > > [http://hudi.apache.org/docs/configurations.html#retainCommits] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] nsivabalan edited a comment on issue #2637: [SUPPORT] - Partial Update : update few columns of a table
nsivabalan edited a comment on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-808563383 @Sugamber : we are currently busy with an upcoming release. Once completed, I will start reviewing this work item. And yes, the linked PRs are similar to your ask. Guess there are few other folks who were interested in this. We can target it for next release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2637: [SUPPORT] - Partial Update : update few columns of a table
nsivabalan commented on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-808563383 @Sugamber : we are currently busy with an upcoming release. Once completed, I will start reviewing this work item. Guess there are few other folks who were interested in this. We can target it for next release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HUDI-1429) NPE HoodieMultiTableDeltaStreamer when using parquet DFS source
[
https://issues.apache.org/jira/browse/HUDI-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan resolved HUDI-1429.
---
Fix Version/s: (was: 0.9.0)
0.8.0
Assignee: sivabalan narayanan
Resolution: Fixed
> NPE HoodieMultiTableDeltaStreamer when using parquet DFS source
> ---
>
> Key: HUDI-1429
> URL: https://issues.apache.org/jira/browse/HUDI-1429
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
>Affects Versions: 0.9.0
>Reporter: Nagabhushanam
>Assignee: sivabalan narayanan
>Priority: Major
> Labels: easyfix, user-support-issues
> Fix For: 0.8.0
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> NPE HoodieMultiTableDeltaStreamer when using parquet DFS source at
> org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java:150
> Below is the error causing snippet
> if
> (cfg.schemaProviderClassName.equals(SchemaRegistryProvider.class.getName())) {
> in case of parquest dfs schema provider is not provided.
> cfg.schemaProviderClassName is null
> so the above should be
> if
> (SchemaRegistryProvider.class.getName().equals(cfg.schemaProviderClassName)) {
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-1429) NPE HoodieMultiTableDeltaStreamer when using parquet DFS source
[
https://issues.apache.org/jira/browse/HUDI-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-1429:
--
Status: In Progress (was: Open)
> NPE HoodieMultiTableDeltaStreamer when using parquet DFS source
> ---
>
> Key: HUDI-1429
> URL: https://issues.apache.org/jira/browse/HUDI-1429
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
>Affects Versions: 0.9.0
>Reporter: Nagabhushanam
>Priority: Major
> Labels: easyfix, user-support-issues
> Fix For: 0.9.0
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> NPE HoodieMultiTableDeltaStreamer when using parquet DFS source at
> org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java:150
> Below is the error causing snippet
> if
> (cfg.schemaProviderClassName.equals(SchemaRegistryProvider.class.getName())) {
> in case of parquest dfs schema provider is not provided.
> cfg.schemaProviderClassName is null
> so the above should be
> if
> (SchemaRegistryProvider.class.getName().equals(cfg.schemaProviderClassName)) {
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-1429) NPE HoodieMultiTableDeltaStreamer when using parquet DFS source
[
https://issues.apache.org/jira/browse/HUDI-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-1429:
--
Status: Open (was: New)
> NPE HoodieMultiTableDeltaStreamer when using parquet DFS source
> ---
>
> Key: HUDI-1429
> URL: https://issues.apache.org/jira/browse/HUDI-1429
> Project: Apache Hudi
> Issue Type: Bug
> Components: DeltaStreamer
>Affects Versions: 0.9.0
>Reporter: Nagabhushanam
>Priority: Major
> Labels: easyfix, user-support-issues
> Fix For: 0.9.0
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> NPE HoodieMultiTableDeltaStreamer when using parquet DFS source at
> org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java:150
> Below is the error causing snippet
> if
> (cfg.schemaProviderClassName.equals(SchemaRegistryProvider.class.getName())) {
> in case of parquest dfs schema provider is not provided.
> cfg.schemaProviderClassName is null
> so the above should be
> if
> (SchemaRegistryProvider.class.getName().equals(cfg.schemaProviderClassName)) {
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] codecov-io edited a comment on pull request #2727: [HOTFIX] fix spark session leak in FunctionalTestHarness
codecov-io edited a comment on pull request #2727: URL: https://github.com/apache/hudi/pull/2727#issuecomment-808296746 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2727?src=pr=h1) Report > Merging [#2727](https://codecov.io/gh/apache/hudi/pull/2727?src=pr=desc) (cd81219) into [release-0.8.0](https://codecov.io/gh/apache/hudi/commit/9bfd810e745202bd27980fa696d61b47922f46a9?el=desc) (9bfd810) will **decrease** coverage by `42.32%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2727?src=pr=tree) ```diff @@ Coverage Diff @@ ## release-0.8.0 #2727 +/- ## === - Coverage51.72% 9.40% -42.33% + Complexity3601 48 -3553 === Files 476 54 -422 Lines225951989-20606 Branches 2409 236 -2173 === - Hits 11687 187-11500 + Misses98891789 -8100 + Partials 1019 13 -1006 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.40% <ø> (-60.34%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2727?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | | | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | | |
[GitHub] [hudi] garyli1019 commented on a change in pull request #2727: [HOTFIX] fix spark session leak in FunctionalTestHarness
garyli1019 commented on a change in pull request #2727:
URL: https://github.com/apache/hudi/pull/2727#discussion_r602388340
##
File path: pom.xml
##
@@ -1424,6 +1425,7 @@
${scala12.version}
2.12
+true
Review comment:
I think we don't need to add `-Dscala2.12` for the spark3 build, since
the scala2.12 was already hardcoded in spark3 profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] garyli1019 commented on a change in pull request #2727: [HOTFIX] fix spark session leak in FunctionalTestHarness
garyli1019 commented on a change in pull request #2727:
URL: https://github.com/apache/hudi/pull/2727#discussion_r602387378
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/FunctionalTestHarness.java
##
@@ -166,5 +168,15 @@ public static synchronized void cleanUpAfterAll() throws
IOException {
for (FileStatus f : fileStatuses) {
fs.delete(f.getPath(), true);
}
+if (spark != null) {
Review comment:
agree. added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] garyli1019 commented on a change in pull request #2727: [HOTFIX] fix spark session leak in FunctionalTestHarness
garyli1019 commented on a change in pull request #2727:
URL: https://github.com/apache/hudi/pull/2727#discussion_r602381959
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -236,6 +236,21 @@ public static void initClass() throws Exception {
prepareParquetDFSFiles(PARQUET_NUM_RECORDS, PARQUET_SOURCE_ROOT);
}
+ @AfterAll
Review comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2727: [HOTFIX] fix spark session leak in FunctionalTestHarness
nsivabalan commented on a change in pull request #2727:
URL: https://github.com/apache/hudi/pull/2727#discussion_r602362761
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/FunctionalTestHarness.java
##
@@ -166,5 +168,15 @@ public static synchronized void cleanUpAfterAll() throws
IOException {
for (FileStatus f : fileStatuses) {
fs.delete(f.getPath(), true);
}
+if (spark != null) {
Review comment:
don't we need to stop hdfsTestService as well?
##
File path: pom.xml
##
@@ -1424,6 +1425,7 @@
${scala12.version}
2.12
+true
Review comment:
whats the expected behavior if we run maven with both spark3 and scala12
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #2727: [HOTFIX] fix spark session leak in FunctionalTestHarness
n3nash commented on a change in pull request #2727:
URL: https://github.com/apache/hudi/pull/2727#discussion_r602377693
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -236,6 +236,21 @@ public static void initClass() throws Exception {
prepareParquetDFSFiles(PARQUET_NUM_RECORDS, PARQUET_SOURCE_ROOT);
}
+ @AfterAll
Review comment:
Can we avoid changes in this class since there are no actual changes ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2727: [HOTFIX] fix spark session leak in FunctionalTestHarness
codecov-io commented on pull request #2727: URL: https://github.com/apache/hudi/pull/2727#issuecomment-808296746 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2727?src=pr=h1) Report > Merging [#2727](https://codecov.io/gh/apache/hudi/pull/2727?src=pr=desc) (9c2b577) into [release-0.8.0](https://codecov.io/gh/apache/hudi/commit/9bfd810e745202bd27980fa696d61b47922f46a9?el=desc) (9bfd810) will **decrease** coverage by `42.32%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2727?src=pr=tree) ```diff @@ Coverage Diff @@ ## release-0.8.0 #2727 +/- ## === - Coverage51.72% 9.40% -42.33% + Complexity3601 48 -3553 === Files 476 54 -422 Lines225951989-20606 Branches 2409 236 -2173 === - Hits 11687 187-11500 + Misses98891789 -8100 + Partials 1019 13 -1006 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.40% <ø> (-60.34%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2727?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | | | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2727/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | | |
[jira] [Closed] (HUDI-1725) Remove AvroSchemaConverter since Flink 1.12.2 already provided
[ https://issues.apache.org/jira/browse/HUDI-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianghu Wang closed HUDI-1725. -- Resolution: Invalid > Remove AvroSchemaConverter since Flink 1.12.2 already provided > -- > > Key: HUDI-1725 > URL: https://issues.apache.org/jira/browse/HUDI-1725 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-1726) Remove AvroSchemaConverter since Flink 1.12.2 already provided
[ https://issues.apache.org/jira/browse/HUDI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianghu Wang closed HUDI-1726. -- Resolution: Invalid > Remove AvroSchemaConverter since Flink 1.12.2 already provided > -- > > Key: HUDI-1726 > URL: https://issues.apache.org/jira/browse/HUDI-1726 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] wangxianghu closed pull request #2726: [HUDI-1726] Remove AvroSchemaConverter since Flink 1.12.2 already pro…
wangxianghu closed pull request #2726: URL: https://github.com/apache/hudi/pull/2726 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #2726: [HUDI-1726] Remove AvroSchemaConverter since Flink 1.12.2 already pro…
wangxianghu commented on pull request #2726: URL: https://github.com/apache/hudi/pull/2726#issuecomment-808177220 > No, i actually keep the class intentionally, because this is a pure tool class and buggy (the code in Flink side was also written by me) and we may need to tweak based on hudi needs, so i would suggest to keep it. ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] MyLanPangzi commented on issue #2717: [SUPPORT] run_sync_tool support hive3.1.2 on hadoop3.1.4
MyLanPangzi commented on issue #2717: URL: https://github.com/apache/hudi/issues/2717#issuecomment-808123853 this is my pr https://github.com/apache/hudi/pull/2719. but i think this that needs some test for hive2 and hive3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #2726: [HUDI-1726] Remove AvroSchemaConverter since Flink 1.12.2 already pro…
danny0405 commented on pull request #2726: URL: https://github.com/apache/hudi/pull/2726#issuecomment-808099009 No, i actually keep the class intentionally, because this is a pure tool class and buggy (the code in Flink side was also written by me) and we may need to tweak based on hudi needs, so i would suggest to keep it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2726: [HUDI-1726] Remove AvroSchemaConverter since Flink 1.12.2 already pro…
codecov-io edited a comment on pull request #2726: URL: https://github.com/apache/hudi/pull/2726#issuecomment-808097639 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2726?src=pr=h1) Report > Merging [#2726](https://codecov.io/gh/apache/hudi/pull/2726?src=pr=desc) (bd67e50) into [master](https://codecov.io/gh/apache/hudi/commit/8b774fe3313757a8b94ca408079327c62b4a664c?el=desc) (8b774fe) will **increase** coverage by `0.16%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/2726?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2726 +/- ## + Coverage 51.73% 51.90% +0.16% + Complexity 3606 3591 -15 Files 476 475 -1 Lines 2261122472 -139 Branches 2410 2375 -35 - Hits 1169711663 -34 + Misses 9891 9799 -92 + Partials 1023 1010 -13 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.92% <ø> (+0.01%)` | `0.00 <ø> (ø)` | | | hudiflink | `55.37% <50.00%> (+1.19%)` | `0.00 <0.00> (ø)` | | | hudihadoopmr | `33.44% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisparkdatasource | `70.87% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisync | `45.58% <ø> (ø)` | `0.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiutilities | `69.78% <ø> (+0.05%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2726?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...rg/apache/hudi/streamer/HoodieFlinkStreamerV2.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zdHJlYW1lci9Ib29kaWVGbGlua1N0cmVhbWVyVjIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...java/org/apache/hudi/table/HoodieTableFactory.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZUZhY3RvcnkuamF2YQ==) | `88.09% <ø> (ø)` | `14.00 <0.00> (ø)` | | | [.../java/org/apache/hudi/table/HoodieTableSource.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNvdXJjZS5qYXZh) | `60.97% <100.00%> (ø)` | `25.00 <0.00> (ø)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.72% <0.00%> (+0.34%)` | `56.00% <0.00%> (+1.00%)` | | | [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `78.12% <0.00%> (+1.56%)` | `26.00% <0.00%> (ø%)` | | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2726: [HUDI-1726] Remove AvroSchemaConverter since Flink 1.12.2 already pro…
codecov-io commented on pull request #2726: URL: https://github.com/apache/hudi/pull/2726#issuecomment-808097639 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2726?src=pr=h1) Report > Merging [#2726](https://codecov.io/gh/apache/hudi/pull/2726?src=pr=desc) (bd67e50) into [master](https://codecov.io/gh/apache/hudi/commit/8b774fe3313757a8b94ca408079327c62b4a664c?el=desc) (8b774fe) will **increase** coverage by `0.25%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/2726?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2726 +/- ## + Coverage 51.73% 51.98% +0.25% + Complexity 3606 3401 -205 Files 476 453 -23 Lines 2261120867-1744 Branches 2410 2220 -190 - Hits 1169710847 -850 + Misses 9891 9085 -806 + Partials 1023 935 -88 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.92% <ø> (+0.01%)` | `0.00 <ø> (ø)` | | | hudiflink | `55.37% <50.00%> (+1.19%)` | `0.00 <0.00> (ø)` | | | hudihadoopmr | `33.44% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisparkdatasource | `70.87% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.78% <ø> (+0.05%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2726?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...rg/apache/hudi/streamer/HoodieFlinkStreamerV2.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zdHJlYW1lci9Ib29kaWVGbGlua1N0cmVhbWVyVjIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...java/org/apache/hudi/table/HoodieTableFactory.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZUZhY3RvcnkuamF2YQ==) | `88.09% <ø> (ø)` | `14.00 <0.00> (ø)` | | | [.../java/org/apache/hudi/table/HoodieTableSource.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNvdXJjZS5qYXZh) | `60.97% <100.00%> (ø)` | `25.00 <0.00> (ø)` | | | [.../org/apache/hudi/hive/HoodieHiveSyncException.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNFeGNlcHRpb24uamF2YQ==) | | | | | [...in/java/org/apache/hudi/hive/SchemaDifference.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2NoZW1hRGlmZmVyZW5jZS5qYXZh) | | | | | [...src/main/java/org/apache/hudi/dla/DLASyncTool.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1zeW5jL2h1ZGktZGxhLXN5bmMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZGxhL0RMQVN5bmNUb29sLmphdmE=) | | | | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | | | | | [...apache/hudi/timeline/service/handlers/Handler.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS10aW1lbGluZS1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RpbWVsaW5lL3NlcnZpY2UvaGFuZGxlcnMvSGFuZGxlci5qYXZh) | | | | | [...in/java/org/apache/hudi/hive/HoodieHiveClient.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZUNsaWVudC5qYXZh) | | | | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | | | | | ... and [17 more](https://codecov.io/gh/apache/hudi/pull/2726/diff?src=pr=tree-more) | | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
[GitHub] [hudi] liijiankang commented on issue #2723: [SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
liijiankang commented on issue #2723: URL: https://github.com/apache/hudi/issues/2723#issuecomment-808081021  Thanks，I will test it .If this exception comes from a cleaner, do we have to set hoodie.clean.async = false? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] garyli1019 opened a new pull request #2727: [HOTFIX] fix spark session leak in FunctionalTestHarness
garyli1019 opened a new pull request #2727: URL: https://github.com/apache/hudi/pull/2727 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #2726: [HUDI-1726] Remove AvroSchemaConverter since Flink 1.12.2 already pro…
wangxianghu commented on pull request #2726: URL: https://github.com/apache/hudi/pull/2726#issuecomment-808072218 @yanghua @danny0405 please take a look when free -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1726) Remove AvroSchemaConverter since Flink 1.12.2 already provided
[ https://issues.apache.org/jira/browse/HUDI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1726: - Labels: pull-request-available (was: ) > Remove AvroSchemaConverter since Flink 1.12.2 already provided > -- > > Key: HUDI-1726 > URL: https://issues.apache.org/jira/browse/HUDI-1726 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] wangxianghu opened a new pull request #2726: [HUDI-1726] Remove AvroSchemaConverter since Flink 1.12.2 already pro…
wangxianghu opened a new pull request #2726: URL: https://github.com/apache/hudi/pull/2726 …vided ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *Remove AvroSchemaConverter, Flink 1.12.2 already provided this tool* ## Brief change log ## Verify this pull request This pull request is already covered by existing tests, such as *TestStreamReadOperator*. ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2723: [SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
bvaradar commented on issue #2723: URL: https://github.com/apache/hudi/issues/2723#issuecomment-808063676 This does look like some exception coming from cleaner. Can you look around the logs to see if there are more stack traces related to this ? Can you try setting hoodie.clean.async=False and see ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
codecov-io edited a comment on pull request #2651: URL: https://github.com/apache/hudi/pull/2651#issuecomment-794945140 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=h1) Report > Merging [#2651](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=desc) (0756526) into [master](https://codecov.io/gh/apache/hudi/commit/ce3e8ec87083ef4cd4f33de39b6697f66ff3f277?el=desc) (ce3e8ec) will **increase** coverage by `0.14%`. > The diff coverage is `70.88%`. [](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2651 +/- ## + Coverage 51.76% 51.90% +0.14% - Complexity 3602 3647 +45 Files 476 479 +3 Lines 2257922827 +248 Branches 2408 2449 +41 + Hits 1168811849 +161 - Misses 9874 9919 +45 - Partials 1017 1059 +42 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.87% <0.00%> (-0.06%)` | `0.00 <0.00> (ø)` | | | hudiflink | `54.18% <ø> (-0.10%)` | `0.00 <ø> (ø)` | | | hudihadoopmr | `33.44% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisparkdatasource | `71.27% <75.51%> (+0.39%)` | `0.00 <26.00> (ø)` | | | hudisync | `45.58% <ø> (-0.12%)` | `0.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiutilities | `69.62% <50.00%> (-0.16%)` | `0.00 <0.00> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...c/main/java/org/apache/hudi/common/fs/FSUtils.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZTVXRpbHMuamF2YQ==) | `47.34% <0.00%> (-0.94%)` | `57.00 <0.00> (ø)` | | | [...rg/apache/hudi/common/table/HoodieTableConfig.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlQ29uZmlnLmphdmE=) | `43.20% <0.00%> (-2.25%)` | `17.00 <0.00> (ø)` | | | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `66.66% <0.00%> (-1.65%)` | `43.00 <0.00> (ø)` | | | [...ecution/datasources/Spark2ParsePartitionUtil.scala](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmsyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2V4ZWN1dGlvbi9kYXRhc291cmNlcy9TcGFyazJQYXJzZVBhcnRpdGlvblV0aWwuc2NhbGE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...ecution/datasources/Spark3ParsePartitionUtil.scala](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmszL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2V4ZWN1dGlvbi9kYXRhc291cmNlcy9TcGFyazNQYXJzZVBhcnRpdGlvblV0aWwuc2NhbGE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [.../main/scala/org/apache/hudi/HoodieSparkUtils.scala](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVNwYXJrVXRpbHMuc2NhbGE=) | `83.33% <33.33%> (-5.56%)` | `0.00 <0.00> (ø)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.60% <50.00%> (-1.12%)` | `54.00 <0.00> (-2.00)` | | | [...src/main/scala/org/apache/hudi/DefaultSource.scala](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0RlZmF1bHRTb3VyY2Uuc2NhbGE=) | `78.78% <67.50%> (-5.36%)` | `31.00 <0.00> (+14.00)` | :arrow_down: | | [...c/main/scala/org/apache/hudi/HoodieFileIndex.scala](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZUZpbGVJbmRleC5zY2FsYQ==) | `79.08% <79.08%> (ø)` |
[GitHub] [hudi] codecov-io edited a comment on pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
codecov-io edited a comment on pull request #2651: URL: https://github.com/apache/hudi/pull/2651#issuecomment-794945140 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=h1) Report > Merging [#2651](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=desc) (0756526) into [master](https://codecov.io/gh/apache/hudi/commit/ce3e8ec87083ef4cd4f33de39b6697f66ff3f277?el=desc) (ce3e8ec) will **decrease** coverage by `1.48%`. > The diff coverage is `15.00%`. [](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2651 +/- ## - Coverage 51.76% 50.27% -1.49% + Complexity 3602 3410 -192 Files 476 440 -36 Lines 2257921055-1524 Branches 2408 2208 -200 - Hits 1168810586-1102 + Misses 9874 9562 -312 + Partials 1017 907 -110 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.87% <0.00%> (-0.06%)` | `0.00 <0.00> (ø)` | | | hudiflink | `54.18% <ø> (-0.10%)` | `0.00 <ø> (ø)` | | | hudihadoopmr | `33.44% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `45.58% <ø> (-0.12%)` | `0.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiutilities | `69.62% <50.00%> (-0.16%)` | `0.00 <0.00> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...c/main/java/org/apache/hudi/common/fs/FSUtils.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZTVXRpbHMuamF2YQ==) | `47.34% <0.00%> (-0.94%)` | `57.00 <0.00> (ø)` | | | [...rg/apache/hudi/common/table/HoodieTableConfig.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlQ29uZmlnLmphdmE=) | `43.20% <0.00%> (-2.25%)` | `17.00 <0.00> (ø)` | | | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `66.66% <0.00%> (-1.65%)` | `43.00 <0.00> (ø)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.60% <50.00%> (-1.12%)` | `54.00 <0.00> (-2.00)` | | | [.../java/org/apache/hudi/table/HoodieTableSource.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNvdXJjZS5qYXZh) | `60.97% <0.00%> (-4.00%)` | `25.00% <0.00%> (-3.00%)` | | | [...in/java/org/apache/hudi/table/HoodieTableSink.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNpbmsuamF2YQ==) | `12.19% <0.00%> (-2.10%)` | `2.00% <0.00%> (ø%)` | | | [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `84.05% <0.00%> (-0.56%)` | `11.00% <0.00%> (+4.00%)` | :arrow_down: | | [...ache/hudi/sink/StreamWriteOperatorCoordinator.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3JDb29yZGluYXRvci5qYXZh) | `68.94% <0.00%> (-0.44%)` | `32.00% <0.00%> (ø%)` | | | [...rg/apache/hudi/streamer/HoodieFlinkStreamerV2.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zdHJlYW1lci9Ib29kaWVGbGlua1N0cmVhbWVyVjIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | | | ... and [35 more](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree-more) | | -- This is an automated message from the Apache Git Service. To
[GitHub] [hudi] codecov-io edited a comment on pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
codecov-io edited a comment on pull request #2651: URL: https://github.com/apache/hudi/pull/2651#issuecomment-794945140 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=h1) Report > Merging [#2651](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=desc) (0756526) into [master](https://codecov.io/gh/apache/hudi/commit/ce3e8ec87083ef4cd4f33de39b6697f66ff3f277?el=desc) (ce3e8ec) will **decrease** coverage by `1.53%`. > The diff coverage is `15.00%`. [](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2651 +/- ## - Coverage 51.76% 50.23% -1.54% + Complexity 3602 3220 -382 Files 476 418 -58 Lines 2257919450-3129 Branches 2408 2053 -355 - Hits 11688 9770-1918 + Misses 9874 8848-1026 + Partials 1017 832 -185 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.87% <0.00%> (-0.06%)` | `0.00 <0.00> (ø)` | | | hudiflink | `54.18% <ø> (-0.10%)` | `0.00 <ø> (ø)` | | | hudihadoopmr | `33.44% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.62% <50.00%> (-0.16%)` | `0.00 <0.00> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...c/main/java/org/apache/hudi/common/fs/FSUtils.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZTVXRpbHMuamF2YQ==) | `47.34% <0.00%> (-0.94%)` | `57.00 <0.00> (ø)` | | | [...rg/apache/hudi/common/table/HoodieTableConfig.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlQ29uZmlnLmphdmE=) | `43.20% <0.00%> (-2.25%)` | `17.00 <0.00> (ø)` | | | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `66.66% <0.00%> (-1.65%)` | `43.00 <0.00> (ø)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.60% <50.00%> (-1.12%)` | `54.00 <0.00> (-2.00)` | | | [.../java/org/apache/hudi/table/HoodieTableSource.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNvdXJjZS5qYXZh) | `60.97% <0.00%> (-4.00%)` | `25.00% <0.00%> (-3.00%)` | | | [...in/java/org/apache/hudi/table/HoodieTableSink.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZVNpbmsuamF2YQ==) | `12.19% <0.00%> (-2.10%)` | `2.00% <0.00%> (ø%)` | | | [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `84.05% <0.00%> (-0.56%)` | `11.00% <0.00%> (+4.00%)` | :arrow_down: | | [...ache/hudi/sink/StreamWriteOperatorCoordinator.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlT3BlcmF0b3JDb29yZGluYXRvci5qYXZh) | `68.94% <0.00%> (-0.44%)` | `32.00% <0.00%> (ø%)` | | | [...rg/apache/hudi/streamer/HoodieFlinkStreamerV2.java](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zdHJlYW1lci9Ib29kaWVGbGlua1N0cmVhbWVyVjIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | | | ... and [55 more](https://codecov.io/gh/apache/hudi/pull/2651/diff?src=pr=tree-more) | | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the
[GitHub] [hudi] bvaradar commented on issue #2724: Incremental query misses rows after delete operation using spark
bvaradar commented on issue #2724: URL: https://github.com/apache/hudi/issues/2724#issuecomment-808060354 @satishkotha : Is this something you can take a look at ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-1726) Remove AvroSchemaConverter since Flink 1.12.2 already provided
Xianghu Wang created HUDI-1726: -- Summary: Remove AvroSchemaConverter since Flink 1.12.2 already provided Key: HUDI-1726 URL: https://issues.apache.org/jira/browse/HUDI-1726 Project: Apache Hudi Issue Type: Improvement Reporter: Xianghu Wang Assignee: Xianghu Wang Fix For: 0.9.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1725) Remove AvroSchemaConverter since Flink 1.12.2 already provided
Xianghu Wang created HUDI-1725: -- Summary: Remove AvroSchemaConverter since Flink 1.12.2 already provided Key: HUDI-1725 URL: https://issues.apache.org/jira/browse/HUDI-1725 Project: Apache Hudi Issue Type: Improvement Reporter: Xianghu Wang Assignee: Xianghu Wang Fix For: 0.9.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-io commented on pull request #2725: Increase tableProperties priority
codecov-io commented on pull request #2725: URL: https://github.com/apache/hudi/pull/2725#issuecomment-808059487 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2725?src=pr=h1) Report > Merging [#2725](https://codecov.io/gh/apache/hudi/pull/2725?src=pr=desc) (352297b) into [master](https://codecov.io/gh/apache/hudi/commit/8b774fe3313757a8b94ca408079327c62b4a664c?el=desc) (8b774fe) will **decrease** coverage by `42.36%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/2725?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2725 +/- ## - Coverage 51.73% 9.36% -42.37% + Complexity 3606 48 -3558 Files 476 54 -422 Lines 226111996-20615 Branches 2410 238 -2172 - Hits 11697 187-11510 + Misses 98911796 -8095 + Partials 1023 13 -1010 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.36% <0.00%> (-60.37%)` | `0.00 <0.00> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2725?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2725/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `0.00% <0.00%> (-78.40%)` | `0.00 <0.00> (-18.00)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2725/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2725/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2725/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2725/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2725/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2725/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2725/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2725/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | | | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2725/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00%
[jira] [Commented] (HUDI-1723) DFSPathSelector skips files with the same modify date when read up to source limit
[ https://issues.apache.org/jira/browse/HUDI-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309282#comment-17309282 ] Raymond Xu commented on HUDI-1723: -- [~vinoth] Related to this, I'm thinking maybe it's time to create another path selector as the default one for deltastreamer say `DFSRangePathSelector` to combine the merits of DatePathSelector and reliable checkpointing logic. The current DFSPathSelector scans the whole source base path every time, could induce high cloud billing costs without obvious warning to users. `DFSRangePathSelector` is highlighting the fact that input path is scanned in a moving-window, and it should be generic enough to handle date range and others. > DFSPathSelector skips files with the same modify date when read up to source > limit > -- > > Key: HUDI-1723 > URL: https://issues.apache.org/jira/browse/HUDI-1723 > Project: Apache Hudi > Issue Type: Bug > Components: DeltaStreamer >Reporter: Raymond Xu >Priority: Critical > Fix For: 0.9.0 > > Attachments: Screen Shot 2021-03-26 at 1.42.42 AM.png > > > org.apache.hudi.utilities.sources.helpers.DFSPathSelector#listEligibleFiles > filters the input files based on last saved checkpoint, which was the > modification date from last read file. However, the last read file's > modification date could be duplicated for multiple files and resulted in > skipping a few of them when reading up to source limit. An illustration is > shown in the attached picture. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] danny0405 edited a comment on issue #2712: [SUPPORT] May I ask how to delete data by Flink SQL
danny0405 edited a comment on issue #2712: URL: https://github.com/apache/hudi/issues/2712#issuecomment-808048273 Thanks @stayrascal , batch delete is not supported yet ~, Flink SQL now supports row level delete through the CDC source into hudi, if say, your format can produce messages with Flink `RowKind`: `INSERT/UPDATE/UPDATE_AFTER/UPDATE_BEFORE`. If you use the flink cdc connector or use the cdc formar like `canal-json`, the `update/delete` would be row-level and very natural. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #2712: [SUPPORT] May I ask how to delete data by Flink SQL
danny0405 commented on issue #2712: URL: https://github.com/apache/hudi/issues/2712#issuecomment-808048273 Thanks @stayrascal , batch delete is not supported now ~, Flink SQL now supports row level delete through the CDC source into hudi, if say, your format can produce messages with Flink `RowKind`: `INSERT/UPDATE/UPDATE_AFTER/UPDATE_BEFORE` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1723) DFSPathSelector skips files with the same modify date when read up to source limit
[ https://issues.apache.org/jira/browse/HUDI-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309273#comment-17309273 ] Raymond Xu commented on HUDI-1723: -- [~vinoth], as we discussed, a quick fix is to ignore source limit and keeps reading until modification date changes. This could still be an issue in cases where lots of data files were written in a short timeframe. We saw S3's last modified date is up to seconds, so it's likely to contain many files with duplicate mod date. Breaking the source limit in that case could lead to OOM. I agree the quick fix would work in most cases. Do you think it's worth working on a more reliable checkpointing logic like mod date + filename? > DFSPathSelector skips files with the same modify date when read up to source > limit > -- > > Key: HUDI-1723 > URL: https://issues.apache.org/jira/browse/HUDI-1723 > Project: Apache Hudi > Issue Type: Bug > Components: DeltaStreamer >Reporter: Raymond Xu >Priority: Critical > Fix For: 0.9.0 > > Attachments: Screen Shot 2021-03-26 at 1.42.42 AM.png > > > org.apache.hudi.utilities.sources.helpers.DFSPathSelector#listEligibleFiles > filters the input files based on last saved checkpoint, which was the > modification date from last read file. However, the last read file's > modification date could be duplicated for multiple files and resulted in > skipping a few of them when reading up to source limit. An illustration is > shown in the attached picture. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1724) run_sync_tool support for hive3.1.2 on hadoop3.1.4
[ https://issues.apache.org/jira/browse/HUDI-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309272#comment-17309272 ] Balaji Varadarajan commented on HUDI-1724: -- [~shivnarayan] : Can you please triage this > run_sync_tool support for hive3.1.2 on hadoop3.1.4 > -- > > Key: HUDI-1724 > URL: https://issues.apache.org/jira/browse/HUDI-1724 > Project: Apache Hudi > Issue Type: Bug > Components: Hive Integration >Reporter: Balaji Varadarajan >Priority: Major > > Context: https://github.com/apache/hudi/issues/2717 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] bvaradar commented on issue #2717: [SUPPORT] run_sync_tool support hive3.1.2 on hadoop3.1.4
bvaradar commented on issue #2717: URL: https://github.com/apache/hudi/issues/2717#issuecomment-808046990 @MyLanPangzi : Thanks for opening the issue along with a workaround. Can you open a PR with this changes. This needs to test with other versions of hive to make sure there is no regression bug. cc @nsivabalan : Opened https://issues.apache.org/jira/browse/HUDI-1724 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-1724) run_sync_tool support for hive3.1.2 on hadoop3.1.4
Balaji Varadarajan created HUDI-1724: Summary: run_sync_tool support for hive3.1.2 on hadoop3.1.4 Key: HUDI-1724 URL: https://issues.apache.org/jira/browse/HUDI-1724 Project: Apache Hudi Issue Type: Bug Components: Hive Integration Reporter: Balaji Varadarajan Context: https://github.com/apache/hudi/issues/2717 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
pengzhiwei2018 edited a comment on pull request #2651: URL: https://github.com/apache/hudi/pull/2651#issuecomment-808036420 > LGTM now. I mostly have just minor comments now which you can address. > > Can you run these unit tests you added once with `-Pspark3` to make sure this is running seamlessly for Spark 3 ? The travis tests right now don't run the tests with Spark 3. Hi @umehrot2 , I have address the comments you have mentioned above. And also have build the project with -Pspark3. There is a method parameter change for `PartitioningUtils.parsePartition` in spark3. So I add an `SparkParsePartitionUtil` interface to adapter both spark2 and spark3, just like the `SparkRowSerDe` does. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2712: [SUPPORT] May I ask how to delete data by Flink SQL
bvaradar commented on issue #2712: URL: https://github.com/apache/hudi/issues/2712#issuecomment-808043186 @danny0405 @yanghua : Can you please help with this question ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-1723) DFSPathSelector skips files with the same modify date when read up to source limit
Raymond Xu created HUDI-1723: Summary: DFSPathSelector skips files with the same modify date when read up to source limit Key: HUDI-1723 URL: https://issues.apache.org/jira/browse/HUDI-1723 Project: Apache Hudi Issue Type: Bug Components: DeltaStreamer Reporter: Raymond Xu Fix For: 0.9.0 Attachments: Screen Shot 2021-03-26 at 1.42.42 AM.png org.apache.hudi.utilities.sources.helpers.DFSPathSelector#listEligibleFiles filters the input files based on last saved checkpoint, which was the modification date from last read file. However, the last read file's modification date could be duplicated for multiple files and resulted in skipping a few of them when reading up to source limit. An illustration is shown in the attached picture. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
pengzhiwei2018 commented on a change in pull request #2651:
URL: https://github.com/apache/hudi/pull/2651#discussion_r602102751
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala
##
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi
+
+import java.util.Properties
+
+import scala.collection.JavaConverters._
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hudi.client.common.HoodieSparkEngineContext
+import org.apache.hudi.common.config.{HoodieMetadataConfig,
SerializableConfiguration}
+import org.apache.hudi.common.engine.HoodieLocalEngineContext
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.model.HoodieBaseFile
+import org.apache.hudi.common.table.{HoodieTableMetaClient,
TableSchemaResolver}
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.spark.api.java.JavaSparkContext
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.{InternalRow, expressions}
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.avro.SchemaConverters
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference,
BoundReference, Expression, InterpretedPredicate}
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.execution.datasources.{FileIndex, FileStatusCache,
NoopCache, PartitionDirectory, PartitionUtils}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.unsafe.types.UTF8String
+
+import scala.collection.mutable
+
+/**
+ * A File Index which support partition prune for hoodie snapshot and
read-optimized
+ * query.
+ * Main steps to get the file list for query:
+ * 1、Load all files and partition values from the table path.
+ * 2、Do the partition prune by the partition filter condition.
+ *
+ * There are 3 cases for this:
+ * 1、If the partition columns size is equal to the actually partition path
level, we
+ * read it as partitioned table.(e.g partition column is "dt", the partition
path is "2021-03-10")
+ *
+ * 2、If the partition columns size is not equal to the partition path level,
but the partition
+ * column size is "1" (e.g. partition column is "dt", but the partition path
is "2021/03/10"
+ * who'es directory level is 3).We can still read it as a partitioned table.
We will mapping the
+ * partition path (e.g. 2021/03/10) to the only partition column (e.g. "dt").
+ *
+ * 3、Else the the partition columns size is not equal to the partition
directory level and the
+ * size is great than "1" (e.g. partition column is "dt,hh", the partition
path is "2021/03/10/12")
+ * , we read it as a None Partitioned table because we cannot know how to
mapping the partition
+ * path with the partition columns in this case.
Review comment:
I have add this to the comment of the `prunePartition` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] youngyangp opened a new pull request #2725: Increase tableProperties priority
youngyangp opened a new pull request #2725: URL: https://github.com/apache/hudi/pull/2725 Increase priority of every table's property -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
pengzhiwei2018 commented on pull request #2651: URL: https://github.com/apache/hudi/pull/2651#issuecomment-808036420 > LGTM now. I mostly have just minor comments now which you can address. > > Can you run these unit tests you added once with `-Pspark3` to make sure this is running seamlessly for Spark 3 ? The travis tests right now don't run the tests with Spark 3. Hi @umehrot2 , I have address the comments you have mentioned above. And also have build the project with -Pspark3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
pengzhiwei2018 commented on a change in pull request #2651:
URL: https://github.com/apache/hudi/pull/2651#discussion_r602095082
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala
##
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi
+
+import java.util.Properties
+
+import scala.collection.JavaConverters._
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hudi.client.common.HoodieSparkEngineContext
+import org.apache.hudi.common.config.{HoodieMetadataConfig,
SerializableConfiguration}
+import org.apache.hudi.common.engine.HoodieLocalEngineContext
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.model.HoodieBaseFile
+import org.apache.hudi.common.table.{HoodieTableMetaClient,
TableSchemaResolver}
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.spark.api.java.JavaSparkContext
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.{InternalRow, expressions}
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.avro.SchemaConverters
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference,
BoundReference, Expression, InterpretedPredicate}
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.execution.datasources.{FileIndex, FileStatusCache,
NoopCache, PartitionDirectory, PartitionUtils}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.unsafe.types.UTF8String
+
+import scala.collection.mutable
+
+/**
+ * A File Index which support partition prune for hoodie snapshot and
read-optimized
+ * query.
+ * Main steps to get the file list for query:
+ * 1、Load all files and partition values from the table path.
+ * 2、Do the partition prune by the partition filter condition.
+ *
+ * There are 3 cases for this:
+ * 1、If the partition columns size is equal to the actually partition path
level, we
+ * read it as partitioned table.(e.g partition column is "dt", the partition
path is "2021-03-10")
+ *
+ * 2、If the partition columns size is not equal to the partition path level,
but the partition
+ * column size is "1" (e.g. partition column is "dt", but the partition path
is "2021/03/10"
+ * who'es directory level is 3).We can still read it as a partitioned table.
We will mapping the
+ * partition path (e.g. 2021/03/10) to the only partition column (e.g. "dt").
+ *
+ * 3、Else the the partition columns size is not equal to the partition
directory level and the
+ * size is great than "1" (e.g. partition column is "dt,hh", the partition
path is "2021/03/10/12")
+ * , we read it as a None Partitioned table because we cannot know how to
mapping the partition
+ * path with the partition columns in this case.
+ */
+case class HoodieFileIndex(
+ spark: SparkSession,
+ metaClient: HoodieTableMetaClient,
+ schemaSpec: Option[StructType],
+ options: Map[String, String],
+ @transient fileStatusCache: FileStatusCache = NoopCache)
+ extends FileIndex with Logging {
+
+ private val basePath = metaClient.getBasePath
+
+ @transient private val queryPath = new Path(options.getOrElse("path",
"'path' option required"))
+ /**
+* Get the schema of the table.
+*/
+ lazy val schema: StructType = schemaSpec.getOrElse({
+val schemaUtil = new TableSchemaResolver(metaClient)
+SchemaConverters.toSqlType(schemaUtil.getTableAvroSchema)
+ .dataType.asInstanceOf[StructType]
+ })
+
+ /**
+* Get the partition schema from the hoodie.properties.
+*/
+ private lazy val _partitionSchemaFromProperties: StructType = {
+val tableConfig = metaClient.getTableConfig
+val partitionColumns = tableConfig.getPartitionColumns
+val nameFieldMap = schema.fields.map(filed => filed.name -> filed).toMap
+
+if (partitionColumns.isPresent) {
+ val partitionFields = partitionColumns.get().map(column =>
+nameFieldMap.getOrElse(column, throw new
IllegalArgumentException(s"Cannot find column: '" +
+
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r602094839 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala ## @@ -79,39 +81,52 @@ class DefaultSource extends RelationProvider val allPaths = path.map(p => Seq(p)).getOrElse(Seq()) ++ readPaths val fs = FSUtils.getFs(allPaths.head, sqlContext.sparkContext.hadoopConfiguration) -val globPaths = HoodieSparkUtils.checkAndGlobPathIfNecessary(allPaths, fs) - -val tablePath = DataSourceUtils.getTablePath(fs, globPaths.toArray) +// Use the HoodieFileIndex only if the 'path' has specified with no "*" contains. Review comment: done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r602094723 ## File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java ## @@ -276,6 +276,16 @@ public static void processFiles(FileSystem fs, String basePathStr, Function
[GitHub] [hudi] Ashwanisr opened a new issue #2724: Incremental query misses rows after delete operation using spark
Ashwanisr opened a new issue #2724:
URL: https://github.com/apache/hudi/issues/2724
spark 2.4.4 runtime
**Upsert function:**
```
def upsert(albumDf: DataFrame, tableName: String, key: String, combineKey:
String) = {
albumDf.write
.format("hudi")
.option(TABLE_TYPE_OPT_KEY, COW_TABLE_TYPE_OPT_VAL)
.option(RECORDKEY_FIELD_OPT_KEY, key)
.option(PRECOMBINE_FIELD_OPT_KEY, combineKey)
.option(TABLE_NAME, tableName)
.option(OPERATION_OPT_KEY, UPSERT_OPERATION_OPT_VAL)
.option("hoodie.upsert.shuffle.parallelism", "2")
.option("hoodie.cleaner.policy","KEEP_LATEST_COMMITS")
.option("hoodie.keep.min.commits","3")
.option("hoodie.keep.max.commits","4")
.option("hoodie.cleaner.commits.retained","2")
.option("hoodie.clean.automatic","true")
.mode(Append)
.save(s"$path")
}
```
Using this upsert function I performed 4 upserts. Table state and files are :
hudi/Album/ 0
hudi/Album/.hoodie/ 0
hudi/Album/.hoodie/.aux/ 0
hudi/Album/.hoodie/.aux/.bootstrap/ 0
hudi/Album/.hoodie/.aux/.bootstrap/.fileids/ 0
hudi/Album/.hoodie/.aux/.bootstrap/.partitions/ 0
hudi/Album/.hoodie/.temp/ 0
hudi/Album/.hoodie/20210325102516.commit 1563
hudi/Album/.hoodie/20210325102516.commit.requested 0
hudi/Album/.hoodie/20210325102516.inflight 976
hudi/Album/.hoodie/20210325103000.commit 1573
hudi/Album/.hoodie/20210325103000.commit.requested 0
hudi/Album/.hoodie/20210325103000.inflight 976
hudi/Album/.hoodie/20210325104112.commit 1576
hudi/Album/.hoodie/20210325104112.commit.requested 0
hudi/Album/.hoodie/20210325104112.inflight 1636
hudi/Album/.hoodie/20210325104451.clean 1473
hudi/Album/.hoodie/20210325104451.clean.inflight 1442
hudi/Album/.hoodie/20210325104451.clean.requested 1442
hudi/Album/.hoodie/20210325104451.commit 1576
hudi/Album/.hoodie/20210325104451.commit.requested 0
hudi/Album/.hoodie/20210325104451.inflight 976
hudi/Album/.hoodie/archived/ 0
hudi/Album/.hoodie/hoodie.properties 226
hudi/Album/default/ 0
hudi/Album/default/.hoodie_partition_metadata 93
hudi/Album/default/1fc8c6f5-8a27-421f-9a1c-f55069522179-0_0-101-491_20210325104112.parquet
434836
hudi/Album/default/1fc8c6f5-8a27-421f-9a1c-f55069522179-0_0-132-517_20210325104451.parquet
434911
hudi/Album/default/1fc8c6f5-8a27-421f-9a1c-f55069522179-0_0-53-50_20210325103000.parquet
434781
Table:
_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|
_hoodie_file_name|albumId| title| tracks|updateDate|
+---++--+--++---+--++--+
| 20210325102516| 20210325102516_0_1| 801|
default|1fc8c6f5-8a27-421...|801| Hail to the Thief| [2+2=5, Backdrifts]|
18231|
| 20210325102516| 20210325102516_0_2| 800|
default|1fc8c6f5-8a27-421...|800| 6 String Theory|[Lay it down, Am ...|
18231|
| 20210325103000| 20210325103000_0_1| 802|
default|1fc8c6f5-8a27-421...|802|Best Of Jazz Blues|[Jumpin' the blue...|
18265|
| 20210325104112| 20210325104112_0_3| 803|
default|1fc8c6f5-8a27-421...|803| 7|[Lay it down, Am ...|
18231|
| 20210325104451| 20210325104451_0_4| 804|
default|1fc8c6f5-8a27-421...|804| 7|[Lay it down, Am ...|
18231|
+---++--+--++---+--++--+
Next I perform delete operation using function
```
def delete(albumDf: DataFrame, tableName: String, key: String, combineKey:
String) = {
albumDf.write
.format("hudi")
.option(TABLE_TYPE_OPT_KEY, COW_TABLE_TYPE_OPT_VAL)
.option(RECORDKEY_FIELD_OPT_KEY, key)
.option(PRECOMBINE_FIELD_OPT_KEY, combineKey)
.option(TABLE_NAME, tableName)
.option(OPERATION_OPT_KEY, "delete")
.option("hoodie.upsert.shuffle.parallelism", "2")
.option("hoodie.cleaner.policy","KEEP_LATEST_COMMITS")
.option("hoodie.keep.min.commits","3")
.option("hoodie.keep.max.commits","4")
.option("hoodie.cleaner.commits.retained","2")
.option("hoodie.clean.automatic","true")
.mode(Append)
.save(s"$path")
}
```
and command as given below:
```
val dataPoint = spark.read.format("hudi").
option(QUERY_TYPE_OPT_KEY, QUERY_TYPE_INCREMENTAL_OPT_VAL).
option(BEGIN_INSTANTTIME_OPT_KEY, "0").
option(END_INSTANTTIME_OPT_KEY, "2022031900").
load(path)
//
[GitHub] [hudi] wangxianghu commented on pull request #1946: [HUDI-1176]Upgrade tp log4j2
wangxianghu commented on pull request #1946: URL: https://github.com/apache/hudi/pull/1946#issuecomment-807983997 Hi @hddong，I found the newest version of `Apache Log4j` is `2.14.1` for now, how about using `2.13.3` instead of `2.11.0` ? Besides, I ran the unit test locally and found some `No appenders could be found for logger (org.apache.hadoop.util.Shell)` warnings, It seems `log4j2.properties` is missed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (HUDI-1495) Bump Flink version to 1.12.0
[ https://issues.apache.org/jira/browse/HUDI-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1495. -- Resolution: Done 8b774fe3313757a8b94ca408079327c62b4a664c > Bump Flink version to 1.12.0 > > > Key: HUDI-1495 > URL: https://issues.apache.org/jira/browse/HUDI-1495 > Project: Apache Hudi > Issue Type: Task > Components: newbie >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: easyfix, pull-request-available > Fix For: 0.9.0 > > > The apache Flink 1.12.0 has be released, upgrade the version to 1.12.0 in > order to adapter new Flink interfaces. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] yanghua merged pull request #2718: [HUDI-1495] Bump Flink version to 1.12.2
yanghua merged pull request #2718: URL: https://github.com/apache/hudi/pull/2718 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated (6e803e0 -> 8b774fe)
This is an automated email from the ASF dual-hosted git repository.
vinoyang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git.
from 6e803e0 Moving to 0.9.0-SNAPSHOT on master branch.
add 8b774fe [HUDI-1495] Bump Flink version to 1.12.2 (#2718)
No new revisions were added by this update.
Summary of changes:
hudi-flink/pom.xml | 14 ++
.../apache/hudi/configuration/FlinkOptions.java| 40 +-
.../hudi/sink/StreamWriteOperatorCoordinator.java | 9 +-
.../hudi/streamer/HoodieFlinkStreamerV2.java | 4 +-
.../org/apache/hudi/table/HoodieTableFactory.java | 78 +--
.../org/apache/hudi/table/HoodieTableSink.java | 117
.../org/apache/hudi/table/HoodieTableSource.java | 151 +++--
.../java/org/apache/hudi/util/StreamerUtil.java| 4 +-
...ry => org.apache.flink.table.factories.Factory} | 0
.../org/apache/hudi/sink/StreamWriteITCase.java| 12 +-
.../sink/TestStreamWriteOperatorCoordinator.java | 6 +-
.../hudi/sink/utils/MockOperatorStateStore.java| 7 -
.../sink/utils/StreamWriteFunctionWrapper.java | 2 +-
.../apache/hudi/table/HoodieDataSourceITCase.java | 3 +-
.../apache/hudi/table/TestHoodieTableFactory.java | 86
.../apache/hudi/table/TestHoodieTableSource.java | 3 +-
.../apache/hudi/table/format/TestInputFormat.java | 5 +-
.../org/apache/hudi/utils/TestConfigurations.java | 3 +-
.../test/java/org/apache/hudi/utils/TestData.java | 2 +-
.../utils/factory/CollectSinkTableFactory.java | 6 +-
.../utils/factory/ContinuousFileSourceFactory.java | 36 ++---
.../hudi/utils/source/ContinuousFileSource.java| 62 ++---
.../org.apache.flink.table.factories.Factory | 1 +
.../org.apache.flink.table.factories.TableFactory | 18 ---
packaging/hudi-flink-bundle/pom.xml| 133 +-
pom.xml| 2 +-
style/checkstyle-suppressions.xml | 1 +
27 files changed, 405 insertions(+), 400 deletions(-)
rename
hudi-flink/src/main/resources/META-INF/services/{org.apache.flink.table.factories.TableFactory
=> org.apache.flink.table.factories.Factory} (100%)
delete mode 100644
hudi-flink/src/test/resources/META-INF/services/org.apache.flink.table.factories.TableFactory
[GitHub] [hudi] liijiankang commented on issue #2723: [SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
liijiankang commented on issue #2723: URL: https://github.com/apache/hudi/issues/2723#issuecomment-807969332 Hudi version :0.6.0 Spark version :2.4.0+cdh6.2.0 Hive version :2.1.1+cdh6.2.0 Hadoop version :3.0.0+cdh6.2.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] liijiankang opened a new issue #2723: [SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
liijiankang opened a new issue #2723: URL: https://github.com/apache/hudi/issues/2723 **Describe the problem you faced** We use Structured Streaming to subscribe to the data in Kafka, and then write the data to the hoodie, the program will stop abnormally after running for a period of time. **Environment Description** * Hudi version : * Spark version : * Hive version : * Hadoop version : * Storage (HDFS/S3/GCS..) :HDFS * Running on Docker? (yes/no) :no ```Add the stacktrace of the error.```    -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
