[GitHub] [hudi] hudi-bot edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
hudi-bot edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-863021428 ## CI report: * 8dddccdd400bc19605abbbad78c2889764cff4c2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=278) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=283) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
vinothchandar commented on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-864356223 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3093: [HUDI-2031] JVM occasionally crashes during compaction when spark speculative execution is enabled
vinothchandar commented on pull request #3093: URL: https://github.com/apache/hudi/pull/3093#issuecomment-864355908 Change looks reasonable. let me think bit more around the scenarios before merging this. Anyway we can write a unit test around this scenario? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated: [MINOR] Fix Javadoc wrong references (#3115)
This is an automated email from the ASF dual-hosted git repository.
vinoth pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 7865da1 [MINOR] Fix Javadoc wrong references (#3115)
7865da1 is described below
commit 7865da1e154b5dd9d3063511179993e20adf0579
Author: Wei
AuthorDate: Sat Jun 19 12:51:54 2021 +0800
[MINOR] Fix Javadoc wrong references (#3115)
---
.../java/org/apache/hudi/table/action/rollback/RollbackUtils.java | 3 +--
.../src/main/java/org/apache/hudi/keygen/RowKeyGeneratorHelper.java | 4 ++--
.../org/apache/hudi/hadoop/realtime/RealtimeUnmergedRecordReader.java | 2 +-
3 files changed, 4 insertions(+), 5 deletions(-)
diff --git
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/RollbackUtils.java
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/RollbackUtils.java
index 8537e27..d213fb1 100644
---
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/RollbackUtils.java
+++
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/RollbackUtils.java
@@ -19,7 +19,6 @@
package org.apache.hudi.table.action.rollback;
import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
@@ -88,7 +87,7 @@ public class RollbackUtils {
/**
* Generate all rollback requests that needs rolling back this action
without actually performing rollback for COW table type.
- * @param fs instance of {@link FileSystem} to use.
+ * @param engineContext instance of {@link HoodieEngineContext} to use.
* @param basePath base path of interest.
* @param config instance of {@link HoodieWriteConfig} to use.
* @return {@link List} of {@link ListingBasedRollbackRequest}s thus
collected.
diff --git
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/RowKeyGeneratorHelper.java
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/RowKeyGeneratorHelper.java
index 6b059db..82a580d 100644
---
a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/RowKeyGeneratorHelper.java
+++
b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/RowKeyGeneratorHelper.java
@@ -135,8 +135,8 @@ public class RowKeyGeneratorHelper {
* 4 =
"StructField(nested_col,StructType(StructField(prop1,StringType,false),
StructField(prop2,LongType,false)),false)"
*
* the logic fetches the value from field nested_col.prop1.
- * If any level of the nested field is null, {@link
NULL_RECORDKEY_PLACEHOLDER} is returned.
- * If the field value is an empty String, {@link
EMPTY_RECORDKEY_PLACEHOLDER} is returned.
+ * If any level of the nested field is null, {@link
KeyGenUtils#NULL_RECORDKEY_PLACEHOLDER} is returned.
+ * If the field value is an empty String, {@link
KeyGenUtils#EMPTY_RECORDKEY_PLACEHOLDER} is returned.
*
* @param row instance of {@link Row} of interest
* @param positions tree style positions where the leaf node need to be
fetched and returned
diff --git
a/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeUnmergedRecordReader.java
b/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeUnmergedRecordReader.java
index d209a5a..9f51e7f 100644
---
a/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeUnmergedRecordReader.java
+++
b/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeUnmergedRecordReader.java
@@ -63,7 +63,7 @@ class RealtimeUnmergedRecordReader extends
AbstractRealtimeRecordReader
* clients to consume.
*
* @param split File split
- * @param jobConf Job Configuration
+ * @param job Job Configuration
* @param realReader Parquet Reader
*/
public RealtimeUnmergedRecordReader(RealtimeSplit split, JobConf job,
[GitHub] [hudi] vinothchandar merged pull request #3115: [MINOR] Fix Javadoc wrong references
vinothchandar merged pull request #3115: URL: https://github.com/apache/hudi/pull/3115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3035: [HUDI-1936] Introduce a optional property for conditional upsert
vinothchandar commented on pull request #3035: URL: https://github.com/apache/hudi/pull/3035#issuecomment-864355261 cc @vingov do you mind taking a review at this, given its a python benefiting change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3115: [MINOR] Fix Javadoc wrong references
codecov-commenter edited a comment on pull request #3115: URL: https://github.com/apache/hudi/pull/3115#issuecomment-864343564 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3110: [HUDI-1248] Increase timeout for deltaStreamerTestRunner in TestHoodi…
vinothchandar commented on pull request #3110: URL: https://github.com/apache/hudi/pull/3110#issuecomment-864354224 does seem like a bandaid :/, may be necessary for now. any idea why that specific value works? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1248) [UMBRELLA] Tests cleanup and fixes
[ https://issues.apache.org/jira/browse/HUDI-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17365844#comment-17365844 ] Raymond Xu commented on HUDI-1248: -- Starting from [this build|https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=204=results], the following tests are consistent failing in Azure CI [ERROR] TestHoodieFileWriterFactory.testGetFileWriter:62 » UnsupportedOperation .orc f... [ERROR] TestHoodieCompactor.testScheduleCompactionWithInflightInstant:155 » IllegalArgument > [UMBRELLA] Tests cleanup and fixes > -- > > Key: HUDI-1248 > URL: https://issues.apache.org/jira/browse/HUDI-1248 > Project: Apache Hudi > Issue Type: Improvement > Components: Testing >Reporter: sivabalan narayanan >Priority: Blocker > Labels: hudi-umbrellas, pull-request-available > Fix For: 0.9.0 > > > There are quite few tickets that requires some fixes to tests. Creating this > umbrella ticket to track all efforts. > > https://issues.apache.org/jira/browse/HUDI-1055 remove .parquet from tests. > https://issues.apache.org/jira/browse/HUDI-1033 ITTestRepairsCommand and > TestRepairsCommand > https://issues.apache.org/jira/browse/HUDI-1010 memory leak. > https://issues.apache.org/jira/browse/HUDI-997 memory leak > https://issues.apache.org/jira/browse/HUDI-664 : Adjust Logging levels to > reduce verbose log msgs in hudi-client > https://issues.apache.org/jira/browse/HUDI-623: Remove > UpgradePayloadFromUberToApache > https://issues.apache.org/jira/browse/HUDI-541: Replace variables/comments > named "data files" to "base file" > https://issues.apache.org/jira/browse/HUDI-347: Fix > TestHoodieClientOnCopyOnWriteStorage Tests with modular private methods > https://issues.apache.org/jira/browse/HUDI-323: Docker demo/integ-test > stdout/stderr output only available on process exit > https://issues.apache.org/jira/browse/HUDI-284: Need Tests for Hudi handling > of schema evolution > https://issues.apache.org/jira/browse/HUDI-154: Enable Rollback case in > HoodieRealtimeRecordReaderTest.testReader > https://issues.apache.org/jira/browse/HUDI-1143 timestamp micros. > https://issues.apache.org/jira/browse/HUDI-1989: flaky tests in > TestHoodieMergeOnReadTable -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1175) Investigate CI test flakiness (hangs)
[ https://issues.apache.org/jira/browse/HUDI-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu resolved HUDI-1175. -- Fix Version/s: 0.9.0 Resolution: Fixed > Investigate CI test flakiness (hangs) > - > > Key: HUDI-1175 > URL: https://issues.apache.org/jira/browse/HUDI-1175 > Project: Apache Hudi > Issue Type: Bug > Components: Code Cleanup >Affects Versions: 0.6.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1175) Investigate CI test flakiness (hangs)
[ https://issues.apache.org/jira/browse/HUDI-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1175: - Status: Open (was: New) > Investigate CI test flakiness (hangs) > - > > Key: HUDI-1175 > URL: https://issues.apache.org/jira/browse/HUDI-1175 > Project: Apache Hudi > Issue Type: Bug > Components: Code Cleanup >Affects Versions: 0.6.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1175) Investigate CI test flakiness (hangs)
[ https://issues.apache.org/jira/browse/HUDI-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-1175: Assignee: sivabalan narayanan > Investigate CI test flakiness (hangs) > - > > Key: HUDI-1175 > URL: https://issues.apache.org/jira/browse/HUDI-1175 > Project: Apache Hudi > Issue Type: Bug > Components: Code Cleanup >Affects Versions: 0.6.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[hudi] branch master updated: [MINOR] Fix wrong package name (#3114)
This is an automated email from the ASF dual-hosted git repository.
leesf pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 5339606 [MINOR] Fix wrong package name (#3114)
5339606 is described below
commit 53396061cc169efc99849406561ec5ec7cf437ad
Author: Wei
AuthorDate: Sat Jun 19 11:50:01 2021 +0800
[MINOR] Fix wrong package name (#3114)
---
.../{common/table/view => exception}/HoodieMetadataException.java | 0
.../test/java/org/apache/hudi/utilities/sources/TestSqlSource.java| 4 +---
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git
a/hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieMetadataException.java
b/hudi-common/src/main/java/org/apache/hudi/exception/HoodieMetadataException.java
similarity index 100%
rename from
hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieMetadataException.java
rename to
hudi-common/src/main/java/org/apache/hudi/exception/HoodieMetadataException.java
diff --git
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestSqlSource.java
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestSqlSource.java
index 1395ae6..9c3d558 100644
---
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestSqlSource.java
+++
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestSqlSource.java
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package org.apache.hudi.utilities.testutils.sources;
+package org.apache.hudi.utilities.sources;
import org.apache.avro.generic.GenericRecord;
import org.apache.hadoop.fs.Path;
@@ -26,8 +26,6 @@ import
org.apache.hudi.common.testutils.HoodieTestDataGenerator;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter;
import org.apache.hudi.utilities.schema.FilebasedSchemaProvider;
-import org.apache.hudi.utilities.sources.InputBatch;
-import org.apache.hudi.utilities.sources.SqlSource;
import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.sql.AnalysisException;
[GitHub] [hudi] leesf merged pull request #3114: [MINOR] Fix wrong package name
leesf merged pull request #3114: URL: https://github.com/apache/hudi/pull/3114 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3114: [MINOR] Fix wrong package name
leesf commented on a change in pull request #3114: URL: https://github.com/apache/hudi/pull/3114#discussion_r654742890 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestSqlSource.java ## @@ -26,8 +26,6 @@ import org.apache.hudi.common.util.Option; import org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter; import org.apache.hudi.utilities.schema.FilebasedSchemaProvider; -import org.apache.hudi.utilities.sources.InputBatch; -import org.apache.hudi.utilities.sources.SqlSource; Review comment: this is weird that it would pass the checkstyle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3115: [MINOR] Fix Javadoc wrong references
codecov-commenter edited a comment on pull request #3115: URL: https://github.com/apache/hudi/pull/3115#issuecomment-864343564 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3115](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f76a848) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `17.66%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3115 +/- ## = - Coverage 45.85% 28.19% -17.67% + Complexity 5269 1218 -4051 = Files 908 368 -540 Lines 3933213956-25376 Branches 4239 1425 -2814 = - Hits 18036 3935-14101 + Misses19451 9736 -9715 + Partials 1845 285 -1560 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.27% <ø> (-8.18%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `56.67% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ache/hudi/table/action/rollback/RollbackUtils.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9yb2xsYmFjay9Sb2xsYmFja1V0aWxzLmphdmE=) | `0.00% <ø> (ø)` | | | [.../org/apache/hudi/keygen/RowKeyGeneratorHelper.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkva2V5Z2VuL1Jvd0tleUdlbmVyYXRvckhlbHBlci5qYXZh) | `67.44% <ø> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3115: [MINOR] Fix Javadoc wrong references
hudi-bot edited a comment on pull request #3115: URL: https://github.com/apache/hudi/pull/3115#issuecomment-864342921 ## CI report: * f76a8482b21f7311a4360331aba4343d9d4352af Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=280) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3115: [MINOR] Fix Javadoc wrong references
codecov-commenter edited a comment on pull request #3115: URL: https://github.com/apache/hudi/pull/3115#issuecomment-864343564 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3115](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f76a848) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `30.33%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3115 +/- ## = - Coverage 45.85% 15.51% -30.34% + Complexity 5269 431 -4838 = Files 908 272 -636 Lines 3933210510-28822 Branches 4239 1073 -3166 = - Hits 18036 1631-16405 + Misses19451 8730-10721 + Partials 1845 149 -1696 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `56.67% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ache/hudi/table/action/rollback/RollbackUtils.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9yb2xsYmFjay9Sb2xsYmFja1V0aWxzLmphdmE=) | `0.00% <ø> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter commented on pull request #3115: [MINOR] Fix Javadoc wrong references
codecov-commenter commented on pull request #3115: URL: https://github.com/apache/hudi/pull/3115#issuecomment-864343564 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3115](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f76a848) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `42.67%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3115 +/- ## - Coverage 45.85% 3.17% -42.68% + Complexity 5269 82 -5187 Files 908 272 -636 Lines 39332 10510-28822 Branches 42391073 -3166 - Hits 18036 334-17702 + Misses19451 10150 -9301 + Partials 1845 26 -1819 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `10.00% <ø> (-46.64%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3115?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ache/hudi/table/action/rollback/RollbackUtils.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9yb2xsYmFjay9Sb2xsYmFja1V0aWxzLmphdmE=) | `0.00% <ø> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3115/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3115: [MINOR] Fix Javadoc wrong references
hudi-bot edited a comment on pull request #3115: URL: https://github.com/apache/hudi/pull/3115#issuecomment-864342921 ## CI report: * f76a8482b21f7311a4360331aba4343d9d4352af Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=280) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3114: [MINOR] Fix wrong package name
hudi-bot edited a comment on pull request #3114: URL: https://github.com/apache/hudi/pull/3114#issuecomment-864338167 ## CI report: * b8f7f02c0bc4b9621f79397f45cd62cd2c1c6ec8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=279) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3114: [MINOR] Fix wrong package name
codecov-commenter edited a comment on pull request #3114: URL: https://github.com/apache/hudi/pull/3114#issuecomment-864338505 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3114](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b8f7f02) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `0.00%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3114 +/- ## - Coverage 45.85% 45.85% -0.01% Complexity 5269 5269 Files 908 909 +1 Lines 3933239336 +4 Branches 4239 4239 Hits 1803618036 - Misses1945119455 +4 Partials 1845 1845 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `30.44% <ø> (ø)` | | | hudicommon | `47.56% <ø> (-0.02%)` | :arrow_down: | | hudiflink | `61.33% <ø> (ø)` | | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `66.52% <ø> (ø)` | | | hudisync | `51.73% <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `56.63% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/exception/HoodieMetadataException.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhjZXB0aW9uL0hvb2RpZU1ldGFkYXRhRXhjZXB0aW9uLmphdmE=) | `0.00% <ø> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=footer_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Last update [cdb9b48...b8f7f02](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=lastupdated_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3114: [MINOR] Fix wrong package name
codecov-commenter edited a comment on pull request #3114: URL: https://github.com/apache/hudi/pull/3114#issuecomment-864338505 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3115: [MINOR] Fix Javadoc wrong references
hudi-bot commented on pull request #3115: URL: https://github.com/apache/hudi/pull/3115#issuecomment-864342921 ## CI report: * f76a8482b21f7311a4360331aba4343d9d4352af UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] chaplinthink opened a new pull request #3115: [MINOR] Fix Javadoc wrong references
chaplinthink opened a new pull request #3115: URL: https://github.com/apache/hudi/pull/3115 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3114: [MINOR] Fix wrong package name
codecov-commenter edited a comment on pull request #3114: URL: https://github.com/apache/hudi/pull/3114#issuecomment-864338505 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3114](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b8f7f02) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `30.34%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3114 +/- ## = - Coverage 45.85% 15.50% -30.35% + Complexity 5269 430 -4839 = Files 908 272 -636 Lines 3933210510-28822 Branches 4239 1073 -3166 = - Hits 18036 1630-16406 + Misses19451 8730-10721 + Partials 1845 150 -1695 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `56.63% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
hudi-bot edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-863021428 ## CI report: * 8dddccdd400bc19605abbbad78c2889764cff4c2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=278) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
codecov-commenter edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-828848333 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2893](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (8dddccd) into [master](https://codecov.io/gh/apache/hudi/commit/b9e28e5292d9b2a4b665c26eeba660437a6a0a45?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b9e28e5) will **increase** coverage by `0.04%`. > The diff coverage is `62.50%`. [](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2893 +/- ## + Coverage 45.79% 45.84% +0.04% - Complexity 5270 5274 +4 Files 909 908 -1 Lines 3939039400 +10 Branches 4244 4253 +9 + Hits 1803918063 +24 + Misses1950819480 -28 - Partials 1843 1857 +14 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `30.44% <ø> (+0.05%)` | :arrow_up: | | hudicommon | `47.57% <25.00%> (-0.01%)` | :arrow_down: | | hudiflink | `61.33% <ø> (+0.48%)` | :arrow_up: | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `66.44% <71.87%> (-0.09%)` | :arrow_down: | | hudisync | `51.73% <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `56.24% <ø> (-0.40%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...c/main/java/org/apache/hudi/common/fs/FSUtils.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZTVXRpbHMuamF2YQ==) | `47.08% <0.00%> (ø)` | | | [...va/org/apache/hudi/metadata/BaseTableMetadata.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvQmFzZVRhYmxlTWV0YWRhdGEuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [.../org/apache/hudi/metadata/HoodieTableMetadata.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllVGFibGVNZXRhZGF0YS5qYXZh) | `0.00% <ø> (ø)` | | | [...c/main/scala/org/apache/hudi/HoodieFileIndex.scala](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZUZpbGVJbmRleC5zY2FsYQ==) | `76.16% <67.79%> (-4.97%)` | :arrow_down: | | [...e/hudi/metadata/FileSystemBackedTableMetadata.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvRmlsZVN5c3RlbUJhY2tlZFRhYmxlTWV0YWRhdGEuamF2YQ==) | `89.13% <75.00%> (-2.98%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
codecov-commenter edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-828848333 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
codecov-commenter edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-828848333 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2893](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (70b70e9) into [master](https://codecov.io/gh/apache/hudi/commit/b9e28e5292d9b2a4b665c26eeba660437a6a0a45?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b9e28e5) will **increase** coverage by `0.06%`. > The diff coverage is `61.98%`. > :exclamation: Current head 70b70e9 differs from pull request most recent head 8dddccd. Consider uploading reports for the commit 8dddccd to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2893 +/- ## + Coverage 45.79% 45.86% +0.06% - Complexity 5270 5276 +6 Files 909 908 -1 Lines 3939039401 +11 Branches 4244 4253 +9 + Hits 1803918071 +32 + Misses1950819472 -36 - Partials 1843 1858 +15 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `30.44% <ø> (+0.05%)` | :arrow_up: | | hudicommon | `47.54% <25.00%> (-0.04%)` | :arrow_down: | | hudiflink | `61.33% <ø> (+0.48%)` | :arrow_up: | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `66.42% <71.13%> (-0.11%)` | :arrow_down: | | hudisync | `51.73% <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `56.63% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...c/main/java/org/apache/hudi/common/fs/FSUtils.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZTVXRpbHMuamF2YQ==) | `47.08% <0.00%> (ø)` | | | [...va/org/apache/hudi/metadata/BaseTableMetadata.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvQmFzZVRhYmxlTWV0YWRhdGEuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [.../org/apache/hudi/metadata/HoodieTableMetadata.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllVGFibGVNZXRhZGF0YS5qYXZh) | `0.00% <ø> (ø)` | | | [...c/main/scala/org/apache/hudi/HoodieFileIndex.scala](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZUZpbGVJbmRleC5zY2FsYQ==) | `75.77% <66.66%> (-5.36%)` | :arrow_down: | | [...e/hudi/metadata/FileSystemBackedTableMetadata.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvRmlsZVN5c3RlbUJhY2tlZFRhYmxlTWV0YWRhdGEuamF2YQ==) | `89.13% <75.00%> (-2.98%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
codecov-commenter edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-828848333 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3114: [MINOR] Fix wrong package name
hudi-bot edited a comment on pull request #3114: URL: https://github.com/apache/hudi/pull/3114#issuecomment-864338167 ## CI report: * b8f7f02c0bc4b9621f79397f45cd62cd2c1c6ec8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=279) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #3114: [MINOR] Fix wrong package name
codecov-commenter commented on pull request #3114: URL: https://github.com/apache/hudi/pull/3114#issuecomment-864338505 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3114](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b8f7f02) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `42.67%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3114 +/- ## - Coverage 45.85% 3.17% -42.68% + Complexity 5269 82 -5187 Files 908 272 -636 Lines 39332 10510-28822 Branches 42391073 -3166 - Hits 18036 334-17702 + Misses19451 10150 -9301 + Partials 1845 26 -1819 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `10.00% <ø> (-46.64%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3114?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3114/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
codecov-commenter edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-828848333 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2893](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (8dddccd) into [master](https://codecov.io/gh/apache/hudi/commit/b9e28e5292d9b2a4b665c26eeba660437a6a0a45?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b9e28e5) will **decrease** coverage by `30.39%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2893 +/- ## = - Coverage 45.79% 15.40% -30.40% + Complexity 5270 427 -4843 = Files 909 272 -637 Lines 3939010510-28880 Branches 4244 1073 -3171 = - Hits 18039 1619-16420 + Misses19508 8742-10766 + Partials 1843 149 -1694 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `56.24% <ø> (-0.40%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot commented on pull request #3114: [MINOR] Fix wrong package name
hudi-bot commented on pull request #3114: URL: https://github.com/apache/hudi/pull/3114#issuecomment-864338167 ## CI report: * b8f7f02c0bc4b9621f79397f45cd62cd2c1c6ec8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] chaplinthink opened a new pull request #3114: [MINOR] Fix wrong package name
chaplinthink opened a new pull request #3114: URL: https://github.com/apache/hudi/pull/3114 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
codecov-commenter edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-828848333 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2893](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (70b70e9) into [master](https://codecov.io/gh/apache/hudi/commit/b9e28e5292d9b2a4b665c26eeba660437a6a0a45?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b9e28e5) will **decrease** coverage by `30.28%`. > The diff coverage is `n/a`. > :exclamation: Current head 70b70e9 differs from pull request most recent head 8dddccd. Consider uploading reports for the commit 8dddccd to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2893 +/- ## = - Coverage 45.79% 15.50% -30.29% + Complexity 5270 430 -4840 = Files 909 272 -637 Lines 3939010510-28880 Branches 4244 1073 -3171 = - Hits 18039 1630-16409 + Misses19508 8730-10778 + Partials 1843 150 -1693 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `56.63% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] jkdll removed a comment on issue #3113: [SUPPORT] Deltastreamer Error when reading AVRO Schema with UNION Types
jkdll removed a comment on issue #3113:
URL: https://github.com/apache/hudi/issues/3113#issuecomment-864336980
I have also tested with the latest version on the `master` branch. Same
error.
Moreover, without `--transformer-class
org.apache.hudi.utilities.transform.FlatteningTransformer` I get this error
within the application logs:
```
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39662 partitionPath=1616684167},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39576 partitionPath=1616419975},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39575 partitionPath=1616419975},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39589 partitionPath=1616426425},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=37234 partitionPath=1616426425},
currentLocation='null', newLocation='null'}
```
I believe is triggered due to UNION structs. I believe the flatten
transformer "fixes" this by flattening the struct, however it is only a
workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] jkdll edited a comment on issue #3113: [SUPPORT] Deltastreamer Error when reading AVRO Schema with UNION Types
jkdll edited a comment on issue #3113:
URL: https://github.com/apache/hudi/issues/3113#issuecomment-864336980
I have also tested with the latest version on the `master` branch. Same
error.
Moreover, without `--transformer-class
org.apache.hudi.utilities.transform.FlatteningTransformer` I get this error
within the application logs:
```
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39662 partitionPath=1616684167},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39576 partitionPath=1616419975},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39575 partitionPath=1616419975},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39589 partitionPath=1616426425},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=37234 partitionPath=1616426425},
currentLocation='null', newLocation='null'}
```
I believe is triggered due to UNION structs. I believe the flatten
transformer "fixes" this by flattening the struct, however it is only a
workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] jkdll commented on issue #3113: [SUPPORT] Deltastreamer Error when reading AVRO Schema with UNION Types
jkdll commented on issue #3113:
URL: https://github.com/apache/hudi/issues/3113#issuecomment-864336980
I have also tested with the latest version on the `master` branch. Same
error.
Moreover, without `--transformer-class
org.apache.hudi.utilities.transform.FlatteningTransformer` I get this error,
which I believe is triggered due to UNION structs. I believe the flatten
transformer "fixes" this by flattening the struct, however it is only a
workaround:
```
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39662 partitionPath=1616684167},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39576 partitionPath=1616419975},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39575 partitionPath=1616419975},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=39589 partitionPath=1616426425},
currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException
21/06/19 01:10:36 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=37234 partitionPath=1616426425},
currentLocation='null', newLocation='null'}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
hudi-bot edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-863021428 ## CI report: * 70b70e9e18dd7717b7567c83f7c07e1c0c6c0086 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=277) * 8dddccdd400bc19605abbbad78c2889764cff4c2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=278) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
codecov-commenter edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-828848333 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2893](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (8dddccd) into [master](https://codecov.io/gh/apache/hudi/commit/b9e28e5292d9b2a4b665c26eeba660437a6a0a45?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b9e28e5) will **decrease** coverage by `42.61%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2893 +/- ## - Coverage 45.79% 3.17% -42.62% + Complexity 5270 82 -5188 Files 909 272 -637 Lines 39390 10510-28880 Branches 42441073 -3171 - Hits 18039 334-17705 + Misses19508 10150 -9358 + Partials 1843 26 -1817 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `10.00% <ø> (-46.64%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
hudi-bot edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-863021428 ## CI report: * 50e2f1bbcffbd2db4f16901ac577302ffeb335b4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=244) * 70b70e9e18dd7717b7567c83f7c07e1c0c6c0086 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=277) * 8dddccdd400bc19605abbbad78c2889764cff4c2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] umehrot2 commented on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
umehrot2 commented on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-864335470 @vinothchandar @pengzhiwei2018 addressed the latest comments. If it looks good to you guys, I can land it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
codecov-commenter edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-828848333 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2893](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (70b70e9) into [master](https://codecov.io/gh/apache/hudi/commit/b9e28e5292d9b2a4b665c26eeba660437a6a0a45?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b9e28e5) will **decrease** coverage by `42.61%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2893 +/- ## - Coverage 45.79% 3.17% -42.62% + Complexity 5270 82 -5188 Files 909 272 -637 Lines 39390 10510-28880 Branches 42441073 -3171 - Hits 18039 334-17705 + Misses19508 10150 -9358 + Partials 1843 26 -1817 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `10.00% <ø> (-46.64%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2893?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2893/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] umehrot2 commented on a change in pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
umehrot2 commented on a change in pull request #2893: URL: https://github.com/apache/hudi/pull/2893#discussion_r654728264 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -107,34 +113,61 @@ case class HoodieFileIndex( } @transient @volatile private var fileSystemView: HoodieTableFileSystemView = _ - @transient @volatile private var cachedAllInputFiles: Array[HoodieBaseFile] = _ + @transient @volatile private var cachedAllInputFiles: Map[PartitionRowPath, Map[HoodieBaseFile, Seq[HoodieLogFile]]] = _ Review comment: Made the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
hudi-bot edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-863021428 ## CI report: * 50e2f1bbcffbd2db4f16901ac577302ffeb335b4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=244) * 70b70e9e18dd7717b7567c83f7c07e1c0c6c0086 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=277) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] umehrot2 commented on a change in pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
umehrot2 commented on a change in pull request #2893:
URL: https://github.com/apache/hudi/pull/2893#discussion_r654728218
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala
##
@@ -151,13 +184,33 @@ case class HoodieFileIndex(
metaClient.reloadActiveTimeline()
val activeInstants =
metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants
fileSystemView = new HoodieTableFileSystemView(metaClient, activeInstants,
allFiles)
-cachedAllInputFiles =
fileSystemView.getLatestBaseFiles.iterator().asScala.toArray
-cachedAllPartitionPaths = partitionFiles.keys.toSeq
-cachedFileSize = cachedAllInputFiles.map(_.getFileLen).sum
+
+(tableType, queryType) match {
+ case (MERGE_ON_READ, QUERY_TYPE_SNAPSHOT_OPT_VAL) =>
+// Fetch and store latest base and log files, and their sizes
+cachedAllInputFiles = partitionFiles.map(p => {
+ val latestSlices =
fileSystemView.getLatestMergedFileSlicesBeforeOrOn(p._1.partitionPath,
activeInstants.lastInstant().get().getTimestamp)
Review comment:
Added check.
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala
##
@@ -107,34 +113,61 @@ case class HoodieFileIndex(
}
@transient @volatile private var fileSystemView: HoodieTableFileSystemView =
_
- @transient @volatile private var cachedAllInputFiles: Array[HoodieBaseFile]
= _
+ @transient @volatile private var cachedAllInputFiles: Map[PartitionRowPath,
Map[HoodieBaseFile, Seq[HoodieLogFile]]] = _
@transient @volatile private var cachedFileSize: Long = 0L
- @transient @volatile private var cachedAllPartitionPaths:
Seq[PartitionRowPath] = _
@volatile private var queryAsNonePartitionedTable: Boolean = _
refresh0()
override def rootPaths: Seq[Path] = queryPath :: Nil
+ /**
+ * Invoked by Spark to fetch list of latest base files per partition.
+ *
+ * @param partitionFilters partition column filters
+ * @param dataFilters data columns filters
+ * @return list of PartitionDirectory containing partition to base files
mapping
+ */
override def listFiles(partitionFilters: Seq[Expression],
dataFilters: Seq[Expression]):
Seq[PartitionDirectory] = {
if (queryAsNonePartitionedTable) { // Read as Non-Partitioned table.
Seq(PartitionDirectory(InternalRow.empty, allFiles))
} else {
// Prune the partition path by the partition filters
- val prunedPartitions = prunePartition(cachedAllPartitionPaths,
partitionFilters)
+ val prunedPartitions = prunePartition(cachedAllInputFiles.keys.toSeq,
partitionFilters)
prunedPartitions.map { partition =>
-val fileStatues =
fileSystemView.getLatestBaseFiles(partition.partitionPath).iterator()
- .asScala.toSeq
- .map(_.getFileStatus)
-PartitionDirectory(partition.values, fileStatues)
+val baseFileStatuses =
cachedAllInputFiles(partition).keys.map(_.getFileStatus).toSeq
+PartitionDirectory(partition.values, baseFileStatuses)
}
}
}
+ /**
+ * Fetch list of latest base files and log files per partition.
+ *
+ * @param partitionFilters partition column filters
+ * @param dataFilters data column filters
+ * @return mapping from string partition paths to its base/log files
+ */
+ def listBaseAndLogFiles(partitionFilters: Seq[Expression], dataFilters:
Seq[Expression]):
+ Map[String, Map[HoodieBaseFile, Seq[HoodieLogFile]]] = {
+if (queryAsNonePartitionedTable) {
+ // Read as Non-Partitioned table.
+ cachedAllInputFiles.map(entry => (entry._1.partitionPath, entry._2))
+} else {
+ // Prune the partition path by the partition filters
+ val prunedPartitions = prunePartition(cachedAllInputFiles.keys.toSeq,
partitionFilters)
+ prunedPartitions.map(partition => {
+(partition.partitionPath, cachedAllInputFiles(partition))
+ }).toMap
+}
+ }
+
override def inputFiles: Array[String] = {
-cachedAllInputFiles.map(_.getFileStatus.getPath.toString)
+cachedAllInputFiles.values.flatten.flatMap(baseLogFilesMapping => {
+ Iterator(baseLogFilesMapping._1.getPath) ++
baseLogFilesMapping._2.map(_.getFileStatus.getPath.toString)
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] umehrot2 commented on a change in pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
umehrot2 commented on a change in pull request #2893:
URL: https://github.com/apache/hudi/pull/2893#discussion_r654728183
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala
##
@@ -151,13 +184,33 @@ case class HoodieFileIndex(
metaClient.reloadActiveTimeline()
val activeInstants =
metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants
fileSystemView = new HoodieTableFileSystemView(metaClient, activeInstants,
allFiles)
-cachedAllInputFiles =
fileSystemView.getLatestBaseFiles.iterator().asScala.toArray
-cachedAllPartitionPaths = partitionFiles.keys.toSeq
-cachedFileSize = cachedAllInputFiles.map(_.getFileLen).sum
+
+(tableType, queryType) match {
+ case (MERGE_ON_READ, QUERY_TYPE_SNAPSHOT_OPT_VAL) =>
+// Fetch and store latest base and log files, and their sizes
+cachedAllInputFiles = partitionFiles.map(p => {
+ val latestSlices =
fileSystemView.getLatestMergedFileSlicesBeforeOrOn(p._1.partitionPath,
activeInstants.lastInstant().get().getTimestamp)
+ val baseAndLogFilesMapping =
latestSlices.iterator().asScala.map(slice => {
+(slice.getBaseFile.get(),
slice.getLogFiles.sorted(HoodieLogFile.getLogFileComparator).iterator().asScala.toSeq)
+ }).toMap
+ (p._1, baseAndLogFilesMapping)
+})
+cachedFileSize =
cachedAllInputFiles.values.flatten.map(baseLogFilesMap => {
+ baseLogFilesMap._1.getFileLen +
baseLogFilesMap._2.map(_.getFileSize).sum
+}).sum
+ case (_, _) =>
Review comment:
Incremental relations do not use the file index yet. As I understand, it
uses the commit files to get the list. We can explore that separately on how to
integrate it with file index. I will file a jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] umehrot2 commented on a change in pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
umehrot2 commented on a change in pull request #2893:
URL: https://github.com/apache/hudi/pull/2893#discussion_r654727713
##
File path:
hudi-common/src/main/java/org/apache/hudi/metadata/FileSystemBackedTableMetadata.java
##
@@ -105,6 +107,24 @@ public FileSystemBackedTableMetadata(HoodieEngineContext
engineContext, Serializ
return partitionPaths;
}
+ @Override
+ public Map getAllFilesInPartitions(List
partitionPaths)
+ throws IOException {
+if (partitionPaths == null || partitionPaths.isEmpty()) {
+ return Collections.emptyMap();
+}
+
+int parallelism = Math.min(DEFAULT_LISTING_PARALLELISM,
partitionPaths.size());
+
+List> partitionToFiles =
engineContext.map(partitionPaths, partitionPathStr -> {
+ Path partitionPath = new Path(partitionPathStr);
+ FileSystem fs = partitionPath.getFileSystem(hadoopConf.get());
+ return Pair.of(partitionPathStr, FSUtils.getAllDataFilesInPartition(fs,
partitionPath));
Review comment:
I see yeah FileStatus is not serializable in Hadoop 2, but has been made
Serializable in Hadoop 3. We should fix this in a separate PR for all methods
by introducing `SerializableFileStatus` similar to Spark
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/HadoopFSUtils.scala#L347.
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala
##
@@ -314,18 +379,15 @@ case class HoodieFileIndex(
case None => pathToFetch.append(partitionRowPath)
}
}
-// Fetch the rest from the file system.
-val fetchedPartition2Files =
- spark.sparkContext.parallelize(pathToFetch, Math.min(pathToFetch.size,
maxListParallelism))
-.map { partitionRowPath =>
- // Here we use a LocalEngineContext to get the files in the
partition.
- // We can do this because the TableMetadata.getAllFilesInPartition
only rely on the
- // hadoopConf of the EngineContext.
- val engineContext = new
HoodieLocalEngineContext(serializableConf.get())
- val filesInPartition = FSUtils.getFilesInPartition(engineContext,
metadataConfig,
- basePath, partitionRowPath.fullPartitionPath(basePath))
- (partitionRowPath, filesInPartition)
-}.collect().map(f => f._1 -> f._2).toMap
+
+var fetchedPartition2Files: Map[PartitionRowPath, Array[FileStatus]] =
Map()
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2893: [HUDI-1371] [HUDI-1893] Support metadata based listing for Spark DataSource and Spark SQL
hudi-bot edited a comment on pull request #2893: URL: https://github.com/apache/hudi/pull/2893#issuecomment-863021428 ## CI report: * 50e2f1bbcffbd2db4f16901ac577302ffeb335b4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=244) * 70b70e9e18dd7717b7567c83f7c07e1c0c6c0086 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3082: [HUDI-1717] Metadata Reader should merge all the un-synced but complete instants from the dataset timeline.
leesf commented on a change in pull request #3082: URL: https://github.com/apache/hudi/pull/3082#discussion_r654725223 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataWriter.java ## @@ -40,4 +41,9 @@ void update(HoodieRestoreMetadata restoreMetadata, String instantTime); void update(HoodieRollbackMetadata rollbackMetadata, String instantTime); + + /** + * Return the timestamp of the latest instant synced to the metadata table. + */ + Option getSyncedInstantTime(); Review comment: `getLatestSyncedInstantTime` is better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] jkdll opened a new issue #3113: [SUPPORT] Deltastreamer Error when reading AVRO Schema with UNION Types
jkdll opened a new issue #3113:
URL: https://github.com/apache/hudi/issues/3113
**Problem**
I am running deltastreamer (spark submit below) with schema registry
provider `SchemaRegistryProvider` and source class `AvroKafkaSource`. I am
reading a schema which contains UNION Avro types (sample below). While running,
the deltastreamer seems to not be capable of reading UNION types which contain
Nulls, with the error: `ERROR Client: Application diagnostics message: User
class threw exception: org.apache.spark.sql.avro.IncompatibleSchemaException:
Unsupported type NULL`.
```
spark-submit \
--master yarn \
--deploy-mode cluster \
--files "/home/workspace/configs/stage/*" \
--packages
org.apache.hudi:hudi-utilities-bundle_2.12:0.7.0,org.apache.spark:spark-avro_2.12:2.4.7
\
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer `ls
/lib/hudi/hudi-utilities-bundle_2.11-0.7.0-amzn-1.jar` \
--source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
--op UPSERT \
--source-ordering-field timestamp \
--table-type COPY_ON_WRITE \
--target-table "$1" \
--target-base-path "s3a://aws-hudi-data/data/stage/data/$1" \
--schemaprovider-class
org.apache.hudi.utilities.schema.SchemaRegistryProvider \
--continuous \
--enable-sync \
--min-sync-interval-seconds 5 \
--hoodie-conf "group.id=test" \
--hoodie-conf "auto.offset.reset=earliest" \
--hoodie-conf "hoodie.datasource.write.recordkey.field=body.id" \
--hoodie-conf
"hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator"
\
--hoodie-conf "hoodie.deltastreamer.source.kafka.topic=$2" \
--hoodie-conf
"hoodie.deltastreamer.schemaprovider.registry.url=https://schema-registry-url.com/subjects/$2-value/versions/latest;
\
--hoodie-conf "hoodie.datasource.write.partitionpath.field=timestamp"
```
**The Schema Contains Fields with UNION Structs and Arrays such as the
following:**
```
{
"name": "Title",
"type": [
"null",
{
"type": "record",
"name": "body",
"namespace": "Title.additional.Payload",
"fields": [
{
"name": "Name",
"type": "string"
},
{
"name": "value",
"type": [
"null",
"string"
]
}
]
}
],
"default": null
}
```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
The deltastreamer is running on the AWS EMR version 5.33. Details of the
distribution are below
* Hudi version : 0.7.0
* Spark version : 2.11.12
* Hive version : 2.37-amzn-4
* Hadoop version : 2.10.1-amzn-1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Stacktrace**
```
21/06/18 23:47:53 ERROR Client: Application diagnostics message: User class
threw exception: org.apache.spark.sql.avro.IncompatibleSchemaException:
Unsupported type NULL
at
org.apache.spark.sql.avro.SchemaConverters$.toSqlTypeHelper(SchemaConverters.scala:130)
at
org.apache.spark.sql.avro.SchemaConverters$$anonfun$1.apply(SchemaConverters.scala:82)
at
org.apache.spark.sql.avro.SchemaConverters$$anonfun$1.apply(SchemaConverters.scala:81)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.spark.sql.avro.SchemaConverters$.toSqlTypeHelper(SchemaConverters.scala:81)
at
org.apache.spark.sql.avro.SchemaConverters$.toSqlType(SchemaConverters.scala:46)
at
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2049: [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows
codecov-commenter edited a comment on pull request #2049: URL: https://github.com/apache/hudi/pull/2049#issuecomment-856079085 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2049](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (7f80674) into [master](https://codecov.io/gh/apache/hudi/commit/b9e28e5292d9b2a4b665c26eeba660437a6a0a45?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b9e28e5) will **decrease** coverage by `17.48%`. > The diff coverage is `81.25%`. [](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2049 +/- ## = - Coverage 45.79% 28.30% -17.49% + Complexity 5270 1233 -4037 = Files 909 372 -537 Lines 3939013988-25402 Branches 4244 1426 -2818 = - Hits 18039 3960-14079 + Misses19508 9741 -9767 + Partials 1843 287 -1556 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.45% <81.25%> (-7.94%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `56.63% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../BulkInsertInternalPartitionerWithRowsFactory.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL2J1bGtpbnNlcnQvQnVsa0luc2VydEludGVybmFsUGFydGl0aW9uZXJXaXRoUm93c0ZhY3RvcnkuamF2YQ==) | `50.00% <50.00%> (ø)` | | | [...ecution/bulkinsert/NonSortPartitionerWithRows.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL2J1bGtpbnNlcnQvTm9uU29ydFBhcnRpdGlvbmVyV2l0aFJvd3MuamF2YQ==) | `80.00% <80.00%> (ø)` | | | [...tion/bulkinsert/GlobalSortPartitionerWithRows.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL2J1bGtpbnNlcnQvR2xvYmFsU29ydFBhcnRpdGlvbmVyV2l0aFJvd3MuamF2YQ==) | `83.33% <83.33%> (ø)` | | | [...ulkinsert/RDDPartitionSortPartitionerWithRows.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL2J1bGtpbnNlcnQvUkREUGFydGl0aW9uU29ydFBhcnRpdGlvbmVyV2l0aFJvd3MuamF2YQ==) | `93.33% <93.33%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2049: [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows
codecov-commenter edited a comment on pull request #2049: URL: https://github.com/apache/hudi/pull/2049#issuecomment-856079085 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2049](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (7f80674) into [master](https://codecov.io/gh/apache/hudi/commit/b9e28e5292d9b2a4b665c26eeba660437a6a0a45?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b9e28e5) will **decrease** coverage by `30.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2049 +/- ## = - Coverage 45.79% 15.50% -30.29% + Complexity 5270 430 -4840 = Files 909 272 -637 Lines 3939010510-28880 Branches 4244 1073 -3171 = - Hits 18039 1630-16409 + Misses19508 8730-10778 + Partials 1843 150 -1693 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `56.63% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2049: [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows
codecov-commenter edited a comment on pull request #2049: URL: https://github.com/apache/hudi/pull/2049#issuecomment-856079085 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2049](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (7f80674) into [master](https://codecov.io/gh/apache/hudi/commit/b9e28e5292d9b2a4b665c26eeba660437a6a0a45?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b9e28e5) will **decrease** coverage by `42.61%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2049 +/- ## - Coverage 45.79% 3.17% -42.62% + Complexity 5270 82 -5188 Files 909 272 -637 Lines 39390 10510-28880 Branches 42441073 -3171 - Hits 18039 334-17705 + Misses19508 10150 -9358 + Partials 1843 26 -1817 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.40%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `10.00% <ø> (-46.64%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2049?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2049/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #2049: [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows
hudi-bot edited a comment on pull request #2049: URL: https://github.com/apache/hudi/pull/2049#issuecomment-864282756 ## CI report: * 7f806744cd2ce83e68298391608fedf5e2a067f3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=276) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2049: [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows
hudi-bot edited a comment on pull request #2049: URL: https://github.com/apache/hudi/pull/2049#issuecomment-864282756 ## CI report: * 7f806744cd2ce83e68298391608fedf5e2a067f3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=276) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #2049: [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows
hudi-bot commented on pull request #2049: URL: https://github.com/apache/hudi/pull/2049#issuecomment-864282756 ## CI report: * 7f806744cd2ce83e68298391608fedf5e2a067f3 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3112: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in SpillableMap
hudi-bot edited a comment on pull request #3112: URL: https://github.com/apache/hudi/pull/3112#issuecomment-864259550 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2038) Rollback pending compaction when schedule new compaction
[ https://issues.apache.org/jira/browse/HUDI-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2038: - Labels: pull-request-available (was: ) > Rollback pending compaction when schedule new compaction > > > Key: HUDI-2038 > URL: https://issues.apache.org/jira/browse/HUDI-2038 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > > Rollback pending compaction when schedule new compaction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] sbernauer opened a new pull request #3111: [WIP] Fix KafkaAvroSchemaDeserializer to not rely on reflection
sbernauer opened a new pull request #3111: URL: https://github.com/apache/hudi/pull/3111 Hi Hudi Team! ## What is the purpose of the pull request We are using Deltastreamer to ingest events from kafka into an S3 store. We had some incident this week after some of our producers switched to a new schema version, but some remained on older schem versions. We stumbled over this PR https://github.com/apache/hudi/pull/2619. With the following settings we can enbale the Custom avro kafka deserializer ``` hoodie.deltastreamer.source.kafka.value.deserializer.class=org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer hoodie.deltastreamer.schemaprovider.class=org.apache.hudi.utilities.schema.FilebasedSchemaProvider ``` Doing so we noticed that KafkaAvroSchemaDeserializer tries instanciating the configured SchemaProvider via reflection. It calls the constructor in https://github.com/apache/hudi/blob/cdb9b48170ef98634babd8954392efb1c1b90fcf/hudi-utilities/src/main/java/org/apache/hudi/utilities/deser/KafkaAvroSchemaDeserializer.java#L55 resulting in ``` Caused by: java.lang.NoSuchMethodException: org.apache.hudi.utilities.schema.SchemaProviderWithPostProcessor.(org.apache.hudi.common.config.TypedProperties) ``` There are multiple problems here * The Class has not the needed constructor * As we are using FilebasedSchemaProvider it calls the wrong class wrapping the FilebasedSchemaProvider * The FilebasedSchemaProvider needs the JavaSparkContext to work correctly. I think we dont have access to the JavaSparkContext in KafkaAvroSchemaDeserializer ## Brief change log This PR modifies the KafkaAvroSchemaDeserializer, so that it does not rely on Reflection to call the SchemaProvider. Instead it uses the normal progam flow to ask the SchemProvider for the sourceSchema. It then passes the sourceSchema as Property to the KafkaAvroSchemaDeserializer so that it can be used for deserialization. Anyway: I wonder if it is a good idea to make the usage of KafkaAvroSchemaDeserializer the default. IMHO it woud make sense. ## Verify this pull request You have to enable the feature with ``` hoodie.deltastreamer.source.kafka.value.deserializer.class=org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer ``` The tests are WIP ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection
codecov-commenter commented on pull request #3111: URL: https://github.com/apache/hudi/pull/3111#issuecomment-864241005 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3111?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3111](https://codecov.io/gh/apache/hudi/pull/3111?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b96d84e) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `26.95%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3111?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3111 +/- ## = - Coverage 45.85% 18.90% -26.96% + Complexity 5269 869 -4400 = Files 908 368 -540 Lines 3933213955-25377 Branches 4239 1426 -2813 = - Hits 18036 2638-15398 + Misses1945111155 -8296 + Partials 1845 162 -1683 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.27% <ø> (-8.18%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `10.00% <0.00%> (-46.64%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3111?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...i/utilities/deser/KafkaAvroSchemaDeserializer.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2Rlc2VyL0thZmthQXZyb1NjaGVtYURlc2VyaWFsaXplci5qYXZh) | `0.00% <0.00%> (-83.34%)` | :arrow_down: | | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.28%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3111/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #2915: [HUDI-251] Adds JDBC source support for DeltaStreamer
hudi-bot edited a comment on pull request #2915: URL: https://github.com/apache/hudi/pull/2915#issuecomment-861653740 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #3083: [HUDI-2016] Fixed bootstrap of Metadata Table when some actions are in progress.
leesf commented on pull request #3083: URL: https://github.com/apache/hudi/pull/3083#issuecomment-864101374 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codope commented on pull request #3110: [HUDI-1248] Increase timeout for deltaStreamerTestRunner in TestHoodi…
codope commented on pull request #3110: URL: https://github.com/apache/hudi/pull/3110#issuecomment-864096063 @n3nash Can you please review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] liujinhui1994 closed pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
liujinhui1994 closed pull request #2438: URL: https://github.com/apache/hudi/pull/2438 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3105: [HUDI-2038] Rollback pending compaction when schedule new compaction
leesf commented on a change in pull request #3105:
URL: https://github.com/apache/hudi/pull/3105#discussion_r654508401
##
File path:
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/compact/FlinkScheduleCompactionActionExecutor.java
##
@@ -72,6 +75,17 @@ protected HoodieCompactionPlan scheduleCompaction() {
boolean compactable =
needCompact(config.getInlineCompactTriggerStrategy());
if (compactable) {
LOG.info("Generating compaction plan for merge on read table " +
config.getBasePath());
+ // roll back the inflight compaction first
+ HoodieTimeline pendingCompactionTimeline =
table.getActiveTimeline().filterPendingCompactionTimeline();
Review comment:
would you please clarify why we need rollback inflight compaction here?
Scheduling different compactions should be ok, is any other constraint in flink
codebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3110: [HUDI-1248] Increase timeout for deltaStreamerTestRunner in TestHoodi…
hudi-bot commented on pull request #3110: URL: https://github.com/apache/hudi/pull/3110#issuecomment-863909633 ## CI report: * fcf4d5c2fe72ad8dfde06236344770426a53ee97 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #3106: [HUDI-2040] Make flink writer as exactly-once by default
codecov-commenter commented on pull request #3106: URL: https://github.com/apache/hudi/pull/3106#issuecomment-863729309 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3106?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3106](https://codecov.io/gh/apache/hudi/pull/3106?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (18000d3) into [master](https://codecov.io/gh/apache/hudi/commit/aa6342c3c9af53e7064546becf987e26534d0764?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (aa6342c) will **decrease** coverage by `39.42%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3106?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3106 +/- ## - Coverage 42.59% 3.17% -39.43% + Complexity 4925 82 -4843 Files 908 272 -636 Lines 39341 10510-28831 Branches 42391073 -3166 - Hits 16759 334-16425 + Misses20860 10150-10710 + Partials 1722 26 -1696 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `10.00% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3106?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3106/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3106/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3106/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3106/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3106/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] yanghua merged pull request #3101: [HUDI-2036] Move the compaction plan scheduling out of flink writer c…
yanghua merged pull request #3101: URL: https://github.com/apache/hudi/pull/3101 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
hudi-bot edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-863310563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2036) Move the compaction plan scheduling out of flink writer coordinator
[ https://issues.apache.org/jira/browse/HUDI-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2036: - Labels: pull-request-available (was: ) > Move the compaction plan scheduling out of flink writer coordinator > --- > > Key: HUDI-2036 > URL: https://issues.apache.org/jira/browse/HUDI-2036 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Since HUDI-1955 was fixed, we can move the scheduling out if the coordinator > to make the coordinator more lightweight. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3110: [HUDI-1248] Increase timeout for deltaStreamerTestRunner in TestHoodi…
hudi-bot edited a comment on pull request #3110: URL: https://github.com/apache/hudi/pull/3110#issuecomment-863909633 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3109: [HUDI-2043] HoodieDefaultTimeline$filterPendingCompactionTImeline() m…
hudi-bot edited a comment on pull request #3109: URL: https://github.com/apache/hudi/pull/3109#issuecomment-863900788 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
liujinhui1994 commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-863968623 deltaSync should reset this(...kafka.checkpoint.type) configuration (similar to how we reset checkpoints) In this way, we may need to store this in the metadata file. If it is a memory modification, there is a greater risk. I have submitted my latest implementation, please help to see if it is feasible @nsivabalan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #2915: [HUDI-251] Adds JDBC source support for DeltaStreamer
codope commented on a change in pull request #2915:
URL: https://github.com/apache/hudi/pull/2915#discussion_r654276684
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JdbcSource.java
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.utilities.SqlQueryBuilder;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.DataFrameReader;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.functions;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.storage.StorageLevel;
+
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Reads data from RDBMS data sources.
+ */
+
+public class JdbcSource extends RowSource {
+
+ private static final Logger LOG = LogManager.getLogger(JdbcSource.class);
+ private static final List DB_LIMIT_CLAUSE = Arrays.asList("mysql",
"postgresql", "h2");
+ private static final String URI_JDBC_PREFIX = "jdbc:";
+
+ public JdbcSource(TypedProperties props, JavaSparkContext sparkContext,
SparkSession sparkSession,
+SchemaProvider schemaProvider) {
+super(props, sparkContext, sparkSession, schemaProvider);
+ }
+
+ /**
+ * Validates all user properties and prepares the {@link DataFrameReader} to
read from RDBMS.
+ *
+ * @param sessionThe {@link SparkSession}.
+ * @param properties The JDBC connection properties and data source options.
+ * @return The {@link DataFrameReader} to read from RDBMS
+ * @throws HoodieException
+ */
+ private static DataFrameReader validatePropsAndGetDataFrameReader(final
SparkSession session,
+final
TypedProperties properties)
+ throws HoodieException {
+DataFrameReader dataFrameReader;
+FSDataInputStream passwordFileStream = null;
+try {
+ dataFrameReader = session.read().format("jdbc");
+ dataFrameReader = dataFrameReader.option(Config.URL_PROP,
properties.getString(Config.URL));
+ dataFrameReader = dataFrameReader.option(Config.USER_PROP,
properties.getString(Config.USER));
+ dataFrameReader = dataFrameReader.option(Config.DRIVER_PROP,
properties.getString(Config.DRIVER_CLASS));
+ dataFrameReader = dataFrameReader
+ .option(Config.RDBMS_TABLE_PROP,
properties.getString(Config.RDBMS_TABLE_NAME));
+
+ if (properties.containsKey(Config.PASSWORD)) {
+LOG.info("Reading JDBC password from properties file");
+dataFrameReader = dataFrameReader.option(Config.PASSWORD_PROP,
properties.getString(Config.PASSWORD));
+ } else if (properties.containsKey(Config.PASSWORD_FILE)
+ &&
!StringUtils.isNullOrEmpty(properties.getString(Config.PASSWORD_FILE))) {
+LOG.info(String.format("Reading JDBC password from password file %s",
properties.getString(Config.PASSWORD_FILE)));
+FileSystem fileSystem =
FileSystem.get(session.sparkContext().hadoopConfiguration());
+passwordFileStream = fileSystem.open(new
Path(properties.getString(Config.PASSWORD_FILE)));
+byte[] bytes = new byte[passwordFileStream.available()];
+passwordFileStream.read(bytes);
+dataFrameReader = dataFrameReader.option(Config.PASSWORD_PROP, new
String(bytes));
+ } else {
+throw new
[GitHub] [hudi] danny0405 opened a new pull request #3106: [HUDI-2040] Make flink writer as exactly-once by default
danny0405 opened a new pull request #3106: URL: https://github.com/apache/hudi/pull/3106 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3111: Fix KafkaAvroSchemaDeserializer to not rely on reflection
codecov-commenter edited a comment on pull request #3111: URL: https://github.com/apache/hudi/pull/3111#issuecomment-864241005 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] dwshmilyss opened a new issue #3107: [SUPPORT]
dwshmilyss opened a new issue #3107: URL: https://github.com/apache/hudi/issues/3107 Use Spark to Hudi and JVM Metaspace OOM - Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? No - Join the mailing list to engage in conversations and get faster support at dev-subscr...@hudi.apache.org. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** When I use the Spark API to write a piece of data to Hudi, I notice that the JVM's Metaspace continues to grow until the OOM.  Here's my code:  Then I let this method loop 1000 times, I found every JVM load of classes named GeneratedSerializationConstructorAccessor, this look like a generated by reflection.  this is my start jvm parameters. ``` -Xmn400m -Xms2000m -Xmx2000m -XX:SurvivorRatio=2 -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=256m -XX:+CMSClassUnloadingEnabled -XX:SoftRefLRUPolicyMSPerMB=1000 -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UnlockDiagnosticVMOptions -XX:+HeapDumpOnOutOfMemoryError -Dsun.reflect.inflationThreshold=2147483647 -XX:+TraceClassLoading -XX:+TraceClassUnloading -XX:HeapDumpPath=/Users/edz/Desktop/heapDump1.hprof ``` and this is number of jvm load classes.  GeneratedSerializationConstructorAccessor loaded by sun.reflect.DelegatingClassLoader.These classes remain loaded until Full GC and are unloaded. This results in frequent Full GC. Trace the code and I find that these reflections are caused by Spark's transform operator, since an operator like map calls sc.clean() at the beginning. In this method, the following method is called.  Can any one please help us to fix this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] rmahindra321 opened a new pull request #3112: Implement RockDbBasedMap as an alternate to DiskBasedMap in SpillableMap
rmahindra321 opened a new pull request #3112: URL: https://github.com/apache/hudi/pull/3112 ## What is the purpose of the pull request This pull request adds a new alternative based on RockDb for the Disk Based Map that is used within the ExternalSpillableMap. Our benchmark results shows that RockDb may improve performance significantly when the data set is large while available memory may be scarce. RockDb supports compression, efficient memory usage and native library, that may be more efficient in certain situations. By default, disk based map will be used, and a config change will be required to enable rocksDb. In this PR, the rocksDB support is only enabled for HoodieMergeHandle, and a subsequent PR will extend it to all consumers of ExternalSpillableMap (tracked here HUDI-2044) ## Brief change log - Adds a new alternative based on RockDb for the Disk Based Map that is used within the ExternalSpillableMap. - The support is currently added only for HoodieMergeHandle ## Verify this pull request This change added tests and can be verified as follows: Added the unit test in TestSpillableRocksDBBasedMap Updated the test for TestExternalSpillableMap -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] minihippo opened a new pull request #3108: [HUDI-2042] Compare the field object directly in OverwriteWithLatestA…
minihippo opened a new pull request #3108: URL: https://github.com/apache/hudi/pull/3108 …vroPayload ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request Compare the field object directly in OverwriteWithLatestAvroPayload ## Brief change log - Modify OverwriteWithLatestAvroPayload ## Verify this pull request This pull request is already covered by existing tests, such as *(please describe tests)*. ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #3110: [HUDI-1248] Increase timeout for deltaStreamerTestRunner in TestHoodi…
codecov-commenter commented on pull request #3110: URL: https://github.com/apache/hudi/pull/3110#issuecomment-864030127 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3110?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3110](https://codecov.io/gh/apache/hudi/pull/3110?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (fcf4d5c) into [master](https://codecov.io/gh/apache/hudi/commit/cdb9b48170ef98634babd8954392efb1c1b90fcf?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cdb9b48) will **decrease** coverage by `42.67%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3110?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3110 +/- ## - Coverage 45.85% 3.17% -42.68% + Complexity 5269 82 -5187 Files 908 272 -636 Lines 39332 10510-28822 Branches 42391073 -3166 - Hits 18036 334-17702 + Misses19451 10150 -9301 + Partials 1845 26 -1819 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `6.72% <ø> (-45.01%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `10.00% <ø> (-46.64%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3110?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3110/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3110/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3110/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3110/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3110/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3110/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=)
[GitHub] [hudi] codope opened a new pull request #3110: [HUDI-1248] Increase timeout for deltaStreamerTestRunner in TestHoodi…
codope opened a new pull request #3110: URL: https://github.com/apache/hudi/pull/3110 …eDeltaStreamer ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request Attempt to fix `TestHoodieDeltaStreamer.testUpsertsMORContinuousModeWithMultipleWriters` flaky test. ## Brief change log Sometimes, the pending compaction timeline has not yet been updated and there is no compaction request. The assertion for "at least N compaction commits" keeps failing until the deltaStreamerTestRunner times out. This diff increases the timeout from 4 minutes to 6 miuntes to workaround the issue. To reproduce, I ran the test 20 times and it failed thrice. I did the same after this change and it always passed. ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3083: [HUDI-2016] Fixed bootstrap of Metadata Table when some actions are in progress.
leesf commented on a change in pull request #3083:
URL: https://github.com/apache/hudi/pull/3083#discussion_r654496714
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/metadata/TestHoodieBackedMetadata.java
##
@@ -120,46 +120,63 @@ public void testDefaultNoMetadataTable() throws Exception
{
assertThrows(TableNotFoundException.class, () ->
HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePath).build());
// Metadata table is not created if disabled by config
+String firstCommitTime = HoodieActiveTimeline.createNewInstantTime();
try (SparkRDDWriteClient client = new SparkRDDWriteClient(engineContext,
getWriteConfig(true, false))) {
- client.startCommitWithTime("001");
- client.insert(jsc.emptyRDD(), "001");
+ client.startCommitWithTime(firstCommitTime);
+ client.insert(jsc.parallelize(dataGen.generateInserts(firstCommitTime,
5)), firstCommitTime);
Review comment:
would we remove calling `syncTableMetadata()` in
`SparkRDDWriteClient#preWrite`? since it will do nothing as there are in
progress instant always.
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/metadata/TestHoodieBackedMetadata.java
##
@@ -191,8 +208,9 @@ public void testOnlyValidPartitionsAdded() throws Exception
{
final HoodieWriteConfig writeConfig = getWriteConfigBuilder(true, true,
false)
.withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(true).withDirectoryFilterRegex(filterDirRegex).build()).build();
-try (SparkRDDWriteClient client = new SparkRDDWriteClient(engineContext,
writeConfig)) {
+try (SparkRDDWriteClient client = new SparkRDDWriteClient(engineContext,
writeConfig, true)) {
Review comment:
called `Deprecated` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3111: [WIP] Fix KafkaAvroSchemaDeserializer to not rely on reflection
hudi-bot commented on pull request #3111: URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252 ## CI report: * 8985d48eb28ad448a6ec6607fd20c3efcc92b3d2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2915: [HUDI-251] Adds JDBC source support for DeltaStreamer
codecov-commenter edited a comment on pull request #2915: URL: https://github.com/apache/hudi/pull/2915#issuecomment-832697076 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2915: [HUDI-251] Adds JDBC source support for DeltaStreamer
nsivabalan commented on a change in pull request #2915:
URL: https://github.com/apache/hudi/pull/2915#discussion_r654115003
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JdbcSource.java
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.sources;
+
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.utilities.SqlQueryBuilder;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.DataFrameReader;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.functions;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.storage.StorageLevel;
+
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Reads data from RDBMS data sources.
+ */
+
+public class JdbcSource extends RowSource {
+
+ private static final Logger LOG = LogManager.getLogger(JdbcSource.class);
+ private static final List DB_LIMIT_CLAUSE = Arrays.asList("mysql",
"postgresql", "h2");
+ private static final String URI_JDBC_PREFIX = "jdbc:";
+
+ public JdbcSource(TypedProperties props, JavaSparkContext sparkContext,
SparkSession sparkSession,
+SchemaProvider schemaProvider) {
+super(props, sparkContext, sparkSession, schemaProvider);
+ }
+
+ /**
+ * Validates all user properties and prepares the {@link DataFrameReader} to
read from RDBMS.
+ *
+ * @param sessionThe {@link SparkSession}.
+ * @param properties The JDBC connection properties and data source options.
+ * @return The {@link DataFrameReader} to read from RDBMS
+ * @throws HoodieException
+ */
+ private static DataFrameReader validatePropsAndGetDataFrameReader(final
SparkSession session,
+final
TypedProperties properties)
+ throws HoodieException {
+DataFrameReader dataFrameReader;
+FSDataInputStream passwordFileStream = null;
+try {
+ dataFrameReader = session.read().format("jdbc");
+ dataFrameReader = dataFrameReader.option(Config.URL_PROP,
properties.getString(Config.URL));
+ dataFrameReader = dataFrameReader.option(Config.USER_PROP,
properties.getString(Config.USER));
+ dataFrameReader = dataFrameReader.option(Config.DRIVER_PROP,
properties.getString(Config.DRIVER_CLASS));
+ dataFrameReader = dataFrameReader
+ .option(Config.RDBMS_TABLE_PROP,
properties.getString(Config.RDBMS_TABLE_NAME));
+
+ if (properties.containsKey(Config.PASSWORD)) {
+LOG.info("Reading JDBC password from properties file");
+dataFrameReader = dataFrameReader.option(Config.PASSWORD_PROP,
properties.getString(Config.PASSWORD));
+ } else if (properties.containsKey(Config.PASSWORD_FILE)
+ &&
!StringUtils.isNullOrEmpty(properties.getString(Config.PASSWORD_FILE))) {
+LOG.info(String.format("Reading JDBC password from password file %s",
properties.getString(Config.PASSWORD_FILE)));
+FileSystem fileSystem =
FileSystem.get(session.sparkContext().hadoopConfiguration());
+passwordFileStream = fileSystem.open(new
Path(properties.getString(Config.PASSWORD_FILE)));
+byte[] bytes = new byte[passwordFileStream.available()];
+passwordFileStream.read(bytes);
+dataFrameReader = dataFrameReader.option(Config.PASSWORD_PROP, new
String(bytes));
+ } else {
+throw new
[GitHub] [hudi] hudi-bot edited a comment on pull request #3106: [HUDI-2040] Make flink writer as exactly-once by default
hudi-bot edited a comment on pull request #3106: URL: https://github.com/apache/hudi/pull/3106#issuecomment-863726728 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3100: [HUDI-2034] Support explicit partition compaction strategy for flink …
codecov-commenter edited a comment on pull request #3100: URL: https://github.com/apache/hudi/pull/3100#issuecomment-862906759 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #3106: [HUDI-2040] Make flink writer as exactly-once by default
yanghua merged pull request #3106: URL: https://github.com/apache/hudi/pull/3106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2833: [HUDI-89] Add configOption & refactor Hudi configuration framework
hudi-bot edited a comment on pull request #2833: URL: https://github.com/apache/hudi/pull/2833#issuecomment-864208731 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2028) Implement RockDbBasedMap as an alternate to DiskBasedMap in SpillableMap
[ https://issues.apache.org/jira/browse/HUDI-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2028: - Labels: pull-request-available (was: ) > Implement RockDbBasedMap as an alternate to DiskBasedMap in SpillableMap > > > Key: HUDI-2028 > URL: https://issues.apache.org/jira/browse/HUDI-2028 > Project: Apache Hudi > Issue Type: Improvement > Components: Performance >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: pull-request-available > > Implement RockDbBasedMap as an alternate to DiskBasedMap in SpillableMap > > RockDb can improve perf due to native code and very efficient compression. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3109: [HUDI-2043] HoodieDefaultTimeline$filterPendingCompactionTImeline() m…
codecov-commenter edited a comment on pull request #3109: URL: https://github.com/apache/hudi/pull/3109#issuecomment-864012960 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3111: [WIP] Fix KafkaAvroSchemaDeserializer to not rely on reflection
hudi-bot edited a comment on pull request #3111: URL: https://github.com/apache/hudi/pull/3111#issuecomment-864054252 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3086: [HUDI-1776] Support AlterCommand For Hoodie
pengzhiwei2018 commented on a change in pull request #3086:
URL: https://github.com/apache/hudi/pull/3086#discussion_r654097033
##
File path:
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestAlterTable.scala
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.types.{LongType, StructField, StructType}
+
+class TestAlterTable extends TestHoodieSqlBase {
+
+ test("Test Alter Table") {
+withTempDir { tmp =>
+ Seq("cow", "mor").foreach { tableType =>
+val tableName = generateTableName
+val tablePath = s"${tmp.getCanonicalPath}/$tableName"
+// Create table
+spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | location '$tablePath'
+ | options (
+ | type = '$tableType',
+ | primaryKey = 'id',
+ | preCombineField = 'ts'
+ | )
+ """.stripMargin)
+// Alter table name.
+val newTableName = s"${tableName}_1"
+spark.sql(s"alter table $tableName rename to $newTableName")
+assertResult(true) (
+ spark.sessionState.catalog.tableExists(new
TableIdentifier(newTableName))
+)
+val hadoopConf = spark.sessionState.newHadoopConf()
+val metaClient = HoodieTableMetaClient.builder().setBasePath(tablePath)
+ .setConf(hadoopConf).build()
+assertResult(newTableName) (
+ metaClient.getTableConfig.getTableName
+)
+
+spark.sql(s"insert into $newTableName values(1, 'a1', 10, 1000)")
+
+// Add table column
+spark.sql(s"alter table $newTableName add columns(ext0 string)")
+val table = spark.sessionState.catalog.getTableMetadata(new
TableIdentifier(newTableName))
+assertResult(Seq("id", "name", "price", "ts", "ext0")) {
+ HoodieSqlUtils.removeMetaFields(table.schema).fields.map(_.name)
+}
+checkAnswer(s"select id, name, price, ts, ext0 from $newTableName")(
+ Seq(1, "a1", 10.0, 1000, null)
+)
+// Alter table column type
+spark.sql(s"alter table $newTableName change column id id bigint")
+assertResult(StructType(Seq(StructField("id", LongType, nullable =
true(
+spark.sql(s"select id from $newTableName").schema)
+
+
+// Insert data to the new table.
+spark.sql(s"insert into $newTableName values(2, 'a2', 12, 1000, 'e0')")
Review comment:
No, the new table's schema has changed, the insert statement must match
the new table's schema. So we can not missing the `ext0` column.
##
File path:
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestAlterTable.scala
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.types.{LongType, StructField, StructType}
+
+class TestAlterTable extends TestHoodieSqlBase {
+
+ test("Test Alter Table") {
+
[GitHub] [hudi] hudi-bot commented on pull request #3109: [HUDI-2043] HoodieDefaultTimeline$filterPendingCompactionTImeline() m…
hudi-bot commented on pull request #3109: URL: https://github.com/apache/hudi/pull/3109#issuecomment-863900788 ## CI report: * caa49e88af43cad36a34fa4c05ccdc8309c0c49d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
codecov-commenter edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-850284847 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] swuferhong opened a new pull request #3109: [HUDI-2043] HoodieDefaultTimeline$filterPendingCompactionTImeline() m…
swuferhong opened a new pull request #3109: URL: https://github.com/apache/hudi/pull/3109 …ethod have wrong filter condition ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request HoodieDefaultTimeline$filterPendingCompactionTImeline() method have wrong filter condition. Pending Mode need to exclude completed instance. ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2042) Compare the field object directly in OverwriteWithLatestAvroPayload
[ https://issues.apache.org/jira/browse/HUDI-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2042: - Labels: pull-request-available (was: ) > Compare the field object directly in OverwriteWithLatestAvroPayload > --- > > Key: HUDI-2042 > URL: https://issues.apache.org/jira/browse/HUDI-2042 > Project: Apache Hudi > Issue Type: Improvement >Reporter: XiaoyuGeng >Assignee: XiaoyuGeng >Priority: Trivial > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
