[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378412#comment-17378412 ] ASF GitHub Bot commented on HUDI-2087: -- yanghua merged pull request #3252: URL: https://github.com/apache/hudi/pull/3252 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support Append only in Flink stream > --- > > Key: HUDI-2087 > URL: https://issues.apache.org/jira/browse/HUDI-2087 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > Attachments: image-2021-07-08-22-04-30-039.png, > image-2021-07-08-22-04-40-018.png > > > It is necessary to support append mode in flink stream, as the data lake > should be able to write log type data as parquet high performance without > merge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[hudi] branch master updated: [HUDI-2087] Support Append only in Flink stream (#3252)
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 783c9cb [HUDI-2087] Support Append only in Flink stream (#3252) 783c9cb is described below commit 783c9cb3693ff7d82cc3d43a4964c64a4da19436 Author: yuzhaojing <32435329+yuzhaoj...@users.noreply.github.com> AuthorDate: Sat Jul 10 14:49:35 2021 +0800 [HUDI-2087] Support Append only in Flink stream (#3252) Co-authored-by: 喻兆靖 --- .../hudi/table/action/commit/BucketType.java | 2 +- .../apache/hudi/client/HoodieFlinkWriteClient.java | 10 ++- .../java/org/apache/hudi/io/FlinkAppendHandle.java | 3 +- .../java/org/apache/hudi/io/FlinkCreateHandle.java | 12 +++ .../commit/BaseFlinkCommitActionExecutor.java | 5 +- .../apache/hudi/configuration/FlinkOptions.java| 6 ++ .../sink/partitioner/BucketAssignFunction.java | 44 +++--- .../hudi/sink/partitioner/BucketAssigner.java | 10 ++- .../apache/hudi/streamer/FlinkStreamerConfig.java | 11 ++- .../org/apache/hudi/table/HoodieTableFactory.java | 6 ++ .../org/apache/hudi/sink/TestWriteCopyOnWrite.java | 93 -- .../org/apache/hudi/sink/TestWriteMergeOnRead.java | 15 .../hudi/sink/TestWriteMergeOnReadWithCompact.java | 20 - .../test/java/org/apache/hudi/utils/TestData.java | 33 14 files changed, 242 insertions(+), 28 deletions(-) diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BucketType.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BucketType.java index 70ee473..e1fd161 100644 --- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BucketType.java +++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BucketType.java @@ -19,5 +19,5 @@ package org.apache.hudi.table.action.commit; public enum BucketType { - UPDATE, INSERT + UPDATE, INSERT, APPEND_ONLY } diff --git a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java index 05e4481..71ca1b6 100644 --- a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java +++ b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java @@ -56,6 +56,7 @@ import org.apache.hudi.table.HoodieTable; import org.apache.hudi.table.HoodieTimelineArchiveLog; import org.apache.hudi.table.MarkerFiles; import org.apache.hudi.table.action.HoodieWriteMetadata; +import org.apache.hudi.table.action.commit.BucketType; import org.apache.hudi.table.action.compact.FlinkCompactHelpers; import org.apache.hudi.table.upgrade.FlinkUpgradeDowngrade; import org.apache.hudi.util.FlinkClientUtil; @@ -408,6 +409,12 @@ public class HoodieFlinkWriteClient extends final HoodieRecordLocation loc = record.getCurrentLocation(); final String fileID = loc.getFileId(); final String partitionPath = record.getPartitionPath(); +// append only mode always use FlinkCreateHandle +if (loc.getInstantTime().equals(BucketType.APPEND_ONLY.name())) { + return new FlinkCreateHandle<>(config, instantTime, table, partitionPath, + fileID, table.getTaskContextSupplier()); +} + if (bucketToHandles.containsKey(fileID)) { MiniBatchHandle lastHandle = (MiniBatchHandle) bucketToHandles.get(fileID); if (lastHandle.shouldReplace()) { @@ -424,7 +431,8 @@ public class HoodieFlinkWriteClient extends if (isDelta) { writeHandle = new FlinkAppendHandle<>(config, instantTime, table, partitionPath, fileID, recordItr, table.getTaskContextSupplier()); -} else if (loc.getInstantTime().equals("I")) { +} else if (loc.getInstantTime().equals(BucketType.INSERT.name()) || loc.getInstantTime().equals(BucketType.APPEND_ONLY.name())) { + // use the same handle for insert bucket writeHandle = new FlinkCreateHandle<>(config, instantTime, table, partitionPath, fileID, table.getTaskContextSupplier()); } else { diff --git a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkAppendHandle.java b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkAppendHandle.java index 987f335..41d0666 100644 --- a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkAppendHandle.java +++ b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkAppendHandle.java @@ -25,6 +25,7 @@ import org.apache.hudi.common.model.HoodieRecordPayload; import org.apache.hudi.config.HoodieWriteConfig; import org.apache.hudi.table.HoodieTable; import org.apache.hudi.table.MarkerFiles; +import org.apache.hudi.table.action.commit.BucketType; imp
[GitHub] [hudi] yanghua merged pull request #3252: [HUDI-2087] Support Append only in Flink stream
yanghua merged pull request #3252: URL: https://github.com/apache/hudi/pull/3252 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] wangxianghu merged pull request #3248: [MINOR] Fix some wrong assert reasons
wangxianghu merged pull request #3248: URL: https://github.com/apache/hudi/pull/3248 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated: [MINOR] Fix some wrong assert reasons (#3248)
This is an automated email from the ASF dual-hosted git repository. wangxianghu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 7c6eebf [MINOR] Fix some wrong assert reasons (#3248) 7c6eebf is described below commit 7c6eebf98cdf4fcd727a7ff15962b918f7bf7aaf Author: vinoyang AuthorDate: Sat Jul 10 14:35:40 2021 +0800 [MINOR] Fix some wrong assert reasons (#3248) --- .../src/test/java/org/apache/hudi/sink/TestWriteCopyOnWrite.java| 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteCopyOnWrite.java b/hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteCopyOnWrite.java index b3338a9..e145076 100644 --- a/hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteCopyOnWrite.java +++ b/hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteCopyOnWrite.java @@ -399,7 +399,7 @@ public class TestWriteCopyOnWrite { Map> dataBuffer = funcWrapper.getDataBuffer(); assertThat("Should have 1 data bucket", dataBuffer.size(), is(1)); -assertThat("3 records expect to flush out as a mini-batch", +assertThat("2 records expect to flush out as a mini-batch", dataBuffer.values().stream().findFirst().map(List::size).orElse(-1), is(2)); @@ -461,7 +461,7 @@ public class TestWriteCopyOnWrite { Map> dataBuffer = funcWrapper.getDataBuffer(); assertThat("Should have 1 data bucket", dataBuffer.size(), is(1)); -assertThat("3 records expect to flush out as a mini-batch", +assertThat("2 records expect to flush out as a mini-batch", dataBuffer.values().stream().findFirst().map(List::size).orElse(-1), is(2)); @@ -525,7 +525,7 @@ public class TestWriteCopyOnWrite { Map> dataBuffer = funcWrapper.getDataBuffer(); assertThat("Should have 1 data bucket", dataBuffer.size(), is(1)); -assertThat("3 records expect to flush out as a mini-batch", +assertThat("2 records expect to flush out as a mini-batch", dataBuffer.values().stream().findFirst().map(List::size).orElse(-1), is(2));
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378409#comment-17378409 ] ASF GitHub Bot commented on HUDI-2087: -- hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * 46074982740aaca1dcd73a0f3d66b9e8ea18 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=834) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support Append only in Flink stream > --- > > Key: HUDI-2087 > URL: https://issues.apache.org/jira/browse/HUDI-2087 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > Attachments: image-2021-07-08-22-04-30-039.png, > image-2021-07-08-22-04-40-018.png > > > It is necessary to support append mode in flink stream, as the data lake > should be able to write log type data as parquet high performance without > merge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * 46074982740aaca1dcd73a0f3d66b9e8ea18 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=834) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378408#comment-17378408 ] ASF GitHub Bot commented on HUDI-2087: -- codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4607498) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **increase** coverage by `0.02%`. > The diff coverage is `56.81%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## + Coverage 47.65% 47.68% +0.02% - Complexity 5505 5517 +12 Files 929 929 Lines 4125841291 +33 Branches 4138 4141 +3 + Hits 1966019688 +28 - Misses1985019858 +8 + Partials 1748 1745 -3 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.57% <0.00%> (-0.02%)` | :arrow_down: | | hudicommon | `48.59% <ø> (ø)` | | | hudiflink | `60.09% <73.52%> (+0.25%)` | :arrow_up: | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `67.32% <ø> (ø)` | | | hudisync | `54.51% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `58.60% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/client/HoodieFlinkWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZUZsaW5rV3JpdGVDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/FlinkAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vRmxpbmtBcHBlbmRIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/FlinkCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vRmxpbmtDcmVhdGVIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...e/action/commit/BaseFlinkCommitActionExecutor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_med
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4607498) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **increase** coverage by `0.02%`. > The diff coverage is `56.81%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## + Coverage 47.65% 47.68% +0.02% - Complexity 5505 5517 +12 Files 929 929 Lines 4125841291 +33 Branches 4138 4141 +3 + Hits 1966019688 +28 - Misses1985019858 +8 + Partials 1748 1745 -3 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.57% <0.00%> (-0.02%)` | :arrow_down: | | hudicommon | `48.59% <ø> (ø)` | | | hudiflink | `60.09% <73.52%> (+0.25%)` | :arrow_up: | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `67.32% <ø> (ø)` | | | hudisync | `54.51% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `58.60% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/client/HoodieFlinkWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZUZsaW5rV3JpdGVDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/FlinkAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vRmxpbmtBcHBlbmRIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/FlinkCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vRmxpbmtDcmVhdGVIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...e/action/commit/BaseFlinkCommitActionExecutor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9CYXNlRmxpbmtDb21taXRBY3R
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378407#comment-17378407 ] ASF GitHub Bot commented on HUDI-2087: -- codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4607498) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **decrease** coverage by `3.57%`. > The diff coverage is `56.81%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## - Coverage 47.65% 44.07% -3.58% + Complexity 5505 4864 -641 Files 929 850 -79 Lines 4125836853-4405 Branches 4138 3458 -680 - Hits 1966016242-3418 + Misses1985019365 -485 + Partials 1748 1246 -502 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.57% <0.00%> (-0.02%)` | :arrow_down: | | hudicommon | `48.59% <ø> (ø)` | | | hudiflink | `60.09% <73.52%> (+0.25%)` | :arrow_up: | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.60% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/client/HoodieFlinkWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZUZsaW5rV3JpdGVDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/FlinkAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vRmxpbmtBcHBlbmRIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/FlinkCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vRmxpbmtDcmVhdGVIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...e/action/commit/BaseFlinkCommitActionExecutor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=refe
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4607498) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **decrease** coverage by `3.57%`. > The diff coverage is `56.81%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## - Coverage 47.65% 44.07% -3.58% + Complexity 5505 4864 -641 Files 929 850 -79 Lines 4125836853-4405 Branches 4138 3458 -680 - Hits 1966016242-3418 + Misses1985019365 -485 + Partials 1748 1246 -502 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.57% <0.00%> (-0.02%)` | :arrow_down: | | hudicommon | `48.59% <ø> (ø)` | | | hudiflink | `60.09% <73.52%> (+0.25%)` | :arrow_up: | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.60% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/client/HoodieFlinkWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZUZsaW5rV3JpdGVDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/FlinkAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vRmxpbmtBcHBlbmRIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/FlinkCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vRmxpbmtDcmVhdGVIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...e/action/commit/BaseFlinkCommitActionExecutor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9CYXNlRmxpbmtDb21taXRBY3Rpb25FeGV
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378404#comment-17378404 ] ASF GitHub Bot commented on HUDI-1860: -- codecov-commenter edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870526141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Samrat Deb >Priority: Major > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
codecov-commenter edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870526141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378403#comment-17378403 ] ASF GitHub Bot commented on HUDI-1860: -- hudi-bot edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870410669 ## CI report: * 026842812197f1d47960d2df5d0d65270c49eb59 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=833) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Samrat Deb >Priority: Major > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
hudi-bot edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870410669 ## CI report: * 026842812197f1d47960d2df5d0d65270c49eb59 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=833) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378402#comment-17378402 ] ASF GitHub Bot commented on HUDI-2087: -- codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4607498) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **decrease** coverage by `20.23%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## = - Coverage 47.65% 27.41% -20.24% + Complexity 5505 1288 -4217 = Files 929 381 -548 Lines 4125815108-26150 Branches 4138 1305 -2833 = - Hits 19660 4142-15518 + Misses1985010667 -9183 + Partials 1748 299 -1449 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.05% <0.00%> (-13.54%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.60% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referra
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4607498) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **decrease** coverage by `20.23%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## = - Coverage 47.65% 27.41% -20.24% + Complexity 5505 1288 -4217 = Files 929 381 -548 Lines 4125815108-26150 Branches 4138 1305 -2833 = - Hits 19660 4142-15518 + Misses1985010667 -9183 + Partials 1748 299 -1449 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.05% <0.00%> (-13.54%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.60% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378400#comment-17378400 ] ASF GitHub Bot commented on HUDI-2087: -- codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4607498) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **decrease** coverage by `44.76%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3252 +/- ## - Coverage 47.65% 2.88% -44.77% + Complexity 5505 85 -5420 Files 929 281 -648 Lines 41258 11620-29638 Branches 4138 952 -3186 - Hits 19660 335-19325 + Misses19850 11259 -8591 + Partials 1748 26 -1722 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.59%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.25% <ø> (-49.32%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378401#comment-17378401 ] ASF GitHub Bot commented on HUDI-1860: -- codecov-commenter edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870526141 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3184](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0268428) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **decrease** coverage by `24.52%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3184/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3184 +/- ## - Coverage 27.40% 2.88% -24.53% + Complexity 1287 85 -1202 Files 381 281 -100 Lines 15108 11622 -3486 Branches 1305 952 -353 - Hits 4141 335 -3806 - Misses10667 11261 +594 + Partials300 26 -274 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <ø> (-21.06%)` | :arrow_down: | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `9.25% <0.00%> (-49.33%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-71.15%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
codecov-commenter edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870526141 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3184](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0268428) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **decrease** coverage by `24.52%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3184/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3184 +/- ## - Coverage 27.40% 2.88% -24.53% + Complexity 1287 85 -1202 Files 381 281 -100 Lines 15108 11622 -3486 Branches 1305 952 -353 - Hits 4141 335 -3806 - Misses10667 11261 +594 + Partials300 26 -274 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <ø> (-21.06%)` | :arrow_down: | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `9.25% <0.00%> (-49.33%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-71.15%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Soft
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4607498) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **decrease** coverage by `44.76%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3252 +/- ## - Coverage 47.65% 2.88% -44.77% + Complexity 5505 85 -5420 Files 929 281 -648 Lines 41258 11620-29638 Branches 4138 952 -3186 - Hits 19660 335-19325 + Misses19850 11259 -8591 + Partials 1748 26 -1722 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.59%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.25% <ø> (-49.32%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378399#comment-17378399 ] ASF GitHub Bot commented on HUDI-2087: -- hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * a71a691abd487276776b998c691a2a939a3f4028 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=832) * 46074982740aaca1dcd73a0f3d66b9e8ea18 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=834) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support Append only in Flink stream > --- > > Key: HUDI-2087 > URL: https://issues.apache.org/jira/browse/HUDI-2087 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > Attachments: image-2021-07-08-22-04-30-039.png, > image-2021-07-08-22-04-40-018.png > > > It is necessary to support append mode in flink stream, as the data lake > should be able to write log type data as parquet high performance without > merge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * a71a691abd487276776b998c691a2a939a3f4028 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=832) * 46074982740aaca1dcd73a0f3d66b9e8ea18 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=834) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378398#comment-17378398 ] ASF GitHub Bot commented on HUDI-2087: -- hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * a71a691abd487276776b998c691a2a939a3f4028 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=832) * 46074982740aaca1dcd73a0f3d66b9e8ea18 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support Append only in Flink stream > --- > > Key: HUDI-2087 > URL: https://issues.apache.org/jira/browse/HUDI-2087 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > Attachments: image-2021-07-08-22-04-30-039.png, > image-2021-07-08-22-04-40-018.png > > > It is necessary to support append mode in flink stream, as the data lake > should be able to write log type data as parquet high performance without > merge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * a71a691abd487276776b998c691a2a939a3f4028 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=832) * 46074982740aaca1dcd73a0f3d66b9e8ea18 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378397#comment-17378397 ] ASF GitHub Bot commented on HUDI-1860: -- hudi-bot edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870410669 ## CI report: * d63d6c59a8ba4226acdb618452716c60b97b891c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=828) * 026842812197f1d47960d2df5d0d65270c49eb59 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=833) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Samrat Deb >Priority: Major > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
hudi-bot edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870410669 ## CI report: * d63d6c59a8ba4226acdb618452716c60b97b891c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=828) * 026842812197f1d47960d2df5d0d65270c49eb59 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=833) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378396#comment-17378396 ] ASF GitHub Bot commented on HUDI-1860: -- hudi-bot edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870410669 ## CI report: * d63d6c59a8ba4226acdb618452716c60b97b891c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=828) * 026842812197f1d47960d2df5d0d65270c49eb59 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Samrat Deb >Priority: Major > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
hudi-bot edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870410669 ## CI report: * d63d6c59a8ba4226acdb618452716c60b97b891c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=828) * 026842812197f1d47960d2df5d0d65270c49eb59 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378394#comment-17378394 ] ASF GitHub Bot commented on HUDI-2087: -- hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * a71a691abd487276776b998c691a2a939a3f4028 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=832) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support Append only in Flink stream > --- > > Key: HUDI-2087 > URL: https://issues.apache.org/jira/browse/HUDI-2087 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > Attachments: image-2021-07-08-22-04-30-039.png, > image-2021-07-08-22-04-40-018.png > > > It is necessary to support append mode in flink stream, as the data lake > should be able to write log type data as parquet high performance without > merge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * a71a691abd487276776b998c691a2a939a3f4028 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=832) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2083) Hudi CLI does not work with S3
[ https://issues.apache.org/jira/browse/HUDI-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378392#comment-17378392 ] ASF GitHub Bot commented on HUDI-2083: -- veenaypatil commented on pull request #3222: URL: https://github.com/apache/hudi/pull/3222#issuecomment-877562333 > Integ test failure does seem related actually ? @vinothchandar yes, it's related but it's not working even if I specify test dependencies. The reason for excluding Hadoop dependencies is to avoid conflicts, I faced this with Hadoop2 (hdp 2.7.3.x) and Hadoop3 while testing. The changes are working properly with hadoop2 and hadoop3 but I am not able to find a clean to fix the IT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Hudi CLI does not work with S3 > -- > > Key: HUDI-2083 > URL: https://issues.apache.org/jira/browse/HUDI-2083 > Project: Apache Hudi > Issue Type: Task > Components: CLI >Reporter: Vinay >Assignee: Vinay >Priority: Major > Labels: pull-request-available > > Hudi CLI gives exception when trying to connect to s3 path > {code:java} > create --path s3://some-bucket/tmp/hudi/test_mor --tableName test_mor_s3 > --tableType MERGE_ON_READ > Failed to get instance of org.apache.hadoop.fs.FileSystem > org.apache.hudi.exception.HoodieIOException: Failed to get instance of > org.apache.hadoop.fs.FileSystem > at org.apache.hudi.common.fs.FSUtils.getFs(FSUtils.java:98) > = > create --path s3a://some-bucket/tmp/hudi/test_mor --tableName test_mor_s3 > --tableType MERGE_ON_READ > Command failed java.lang.RuntimeException: java.lang.ClassNotFoundException: > Class org.apache.hadoop.fs.s3a.S3AFileSystem not found > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.s3a.S3AFileSystem not found > java.lang.RuntimeException: java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.s3a.S3AFileSystem not found > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) > at > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654) > {code} > This could be because target/lib folder does not contain hadoop-aws or aws-s3 > dependency. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2083) Hudi CLI does not work with S3
[ https://issues.apache.org/jira/browse/HUDI-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378393#comment-17378393 ] ASF GitHub Bot commented on HUDI-2083: -- veenaypatil edited a comment on pull request #3222: URL: https://github.com/apache/hudi/pull/3222#issuecomment-877562333 > Integ test failure does seem related actually ? @vinothchandar yes, it's related but it's not working even if I specify test dependencies. The reason for excluding Hadoop dependencies is to avoid conflicts, I faced this with Hadoop2 (hdp 2.7.3.x) and Hadoop3 while testing. The changes are working properly with hadoop2 and hadoop3 but I am not able to find a clean way to fix the IT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Hudi CLI does not work with S3 > -- > > Key: HUDI-2083 > URL: https://issues.apache.org/jira/browse/HUDI-2083 > Project: Apache Hudi > Issue Type: Task > Components: CLI >Reporter: Vinay >Assignee: Vinay >Priority: Major > Labels: pull-request-available > > Hudi CLI gives exception when trying to connect to s3 path > {code:java} > create --path s3://some-bucket/tmp/hudi/test_mor --tableName test_mor_s3 > --tableType MERGE_ON_READ > Failed to get instance of org.apache.hadoop.fs.FileSystem > org.apache.hudi.exception.HoodieIOException: Failed to get instance of > org.apache.hadoop.fs.FileSystem > at org.apache.hudi.common.fs.FSUtils.getFs(FSUtils.java:98) > = > create --path s3a://some-bucket/tmp/hudi/test_mor --tableName test_mor_s3 > --tableType MERGE_ON_READ > Command failed java.lang.RuntimeException: java.lang.ClassNotFoundException: > Class org.apache.hadoop.fs.s3a.S3AFileSystem not found > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.s3a.S3AFileSystem not found > java.lang.RuntimeException: java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.s3a.S3AFileSystem not found > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) > at > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654) > {code} > This could be because target/lib folder does not contain hadoop-aws or aws-s3 > dependency. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] veenaypatil commented on pull request #3222: [HUDI-2083] Support Hudi Cli to work with S3
veenaypatil commented on pull request #3222: URL: https://github.com/apache/hudi/pull/3222#issuecomment-877562333 > Integ test failure does seem related actually ? @vinothchandar yes, it's related but it's not working even if I specify test dependencies. The reason for excluding Hadoop dependencies is to avoid conflicts, I faced this with Hadoop2 (hdp 2.7.3.x) and Hadoop3 while testing. The changes are working properly with hadoop2 and hadoop3 but I am not able to find a clean to fix the IT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] veenaypatil edited a comment on pull request #3222: [HUDI-2083] Support Hudi Cli to work with S3
veenaypatil edited a comment on pull request #3222: URL: https://github.com/apache/hudi/pull/3222#issuecomment-877562333 > Integ test failure does seem related actually ? @vinothchandar yes, it's related but it's not working even if I specify test dependencies. The reason for excluding Hadoop dependencies is to avoid conflicts, I faced this with Hadoop2 (hdp 2.7.3.x) and Hadoop3 while testing. The changes are working properly with hadoop2 and hadoop3 but I am not able to find a clean way to fix the IT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a71a691) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **not change** coverage. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@Coverage Diff@@ ## master#3252 +/- ## = Coverage 15.62% 15.62% Complexity 489 489 = Files 281 281 Lines 1162011620 Branches952 952 = Hits 1816 1816 Misses 9643 9643 Partials161 161 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <0.00%> (ø)` | | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `58.57% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3b2a4f2...a71a691](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378391#comment-17378391 ] ASF GitHub Bot commented on HUDI-2087: -- codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a71a691) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **not change** coverage. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@Coverage Diff@@ ## master#3252 +/- ## = Coverage 15.62% 15.62% Complexity 489 489 = Files 281 281 Lines 1162011620 Branches952 952 = Hits 1816 1816 Misses 9643 9643 Partials161 161 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <0.00%> (ø)` | | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `58.57% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3b2a4f2...a71a691](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support Append only in Flink stream > --- > > Key: HUDI-2087 > URL: https://issues.apache.org/jira/browse/HUDI-2087 >
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3248: [MINOR] Fix some wrong assert reasons
codecov-commenter edited a comment on pull request #3248: URL: https://github.com/apache/hudi/pull/3248#issuecomment-877187703 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3248](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (64ad40a) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **increase** coverage by `0.00%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3248/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@Coverage Diff@@ ## master#3248 +/- ## = Coverage 27.40% 27.41% - Complexity 1287 1288+1 = Files 381 381 Lines 1510815108 Branches 1305 1305 = + Hits 4141 4142+1 Misses1066710667 + Partials300 299-1 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `21.05% <ø> (ø)` | | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `58.60% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3248/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.47% <0.00%> (+0.33%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3715267...64ad40a](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378390#comment-17378390 ] ASF GitHub Bot commented on HUDI-2087: -- codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a71a691) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **not change** coverage. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff@@ ## master #3252 +/- ## Coverage 2.88% 2.88% Complexity 85 85 Files 281 281 Lines 11620 11620 Branches952 952 Hits335 335 Misses11259 11259 Partials 26 26 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <0.00%> (ø)` | | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `9.25% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3b2a4f2...a71a691](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support Append only in Flink stream > --- > > Key: HUDI-2087 > URL: https://issues.apache.org/jira/browse/HUDI-2087 > Project: Apach
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a71a691) into [master](https://codecov.io/gh/apache/hudi/commit/3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b2a4f2) will **not change** coverage. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff@@ ## master #3252 +/- ## Coverage 2.88% 2.88% Complexity 85 85 Files 281 281 Lines 11620 11620 Branches952 952 Hits335 335 Misses11259 11259 Partials 26 26 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <0.00%> (ø)` | | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `9.25% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3b2a4f2...a71a691](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378388#comment-17378388 ] ASF GitHub Bot commented on HUDI-2087: -- hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * 701c28f6701201382ccdb911662a26b445595833 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=825) * a71a691abd487276776b998c691a2a939a3f4028 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=832) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support Append only in Flink stream > --- > > Key: HUDI-2087 > URL: https://issues.apache.org/jira/browse/HUDI-2087 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > Attachments: image-2021-07-08-22-04-30-039.png, > image-2021-07-08-22-04-40-018.png > > > It is necessary to support append mode in flink stream, as the data lake > should be able to write log type data as parquet high performance without > merge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3248: [MINOR] Fix some wrong assert reasons
codecov-commenter edited a comment on pull request #3248: URL: https://github.com/apache/hudi/pull/3248#issuecomment-877187703 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3248](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (64ad40a) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **increase** coverage by `0.00%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3248/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@Coverage Diff@@ ## master#3248 +/- ## = Coverage 27.40% 27.41% - Complexity 1287 1288+1 = Files 381 381 Lines 1510815108 Branches 1305 1305 = + Hits 4141 4142+1 Misses1066710667 + Partials300 299-1 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `21.05% <ø> (ø)` | | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `58.60% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3248/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.47% <0.00%> (+0.33%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3715267...64ad40a](https://codecov.io/gh/apache/hudi/pull/3248?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * 701c28f6701201382ccdb911662a26b445595833 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=825) * a71a691abd487276776b998c691a2a939a3f4028 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=832) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378387#comment-17378387 ] ASF GitHub Bot commented on HUDI-2087: -- hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * 701c28f6701201382ccdb911662a26b445595833 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=825) * a71a691abd487276776b998c691a2a939a3f4028 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support Append only in Flink stream > --- > > Key: HUDI-2087 > URL: https://issues.apache.org/jira/browse/HUDI-2087 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > Attachments: image-2021-07-08-22-04-30-039.png, > image-2021-07-08-22-04-40-018.png > > > It is necessary to support append mode in flink stream, as the data lake > should be able to write log type data as parquet high performance without > merge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
hudi-bot edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877374804 ## CI report: * 701c28f6701201382ccdb911662a26b445595833 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=825) * a71a691abd487276776b998c691a2a939a3f4028 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378383#comment-17378383 ] ASF GitHub Bot commented on HUDI-2029: -- codecov-commenter edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865384115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement compression for DiskBasedMap in Spillable Map > --- > > Key: HUDI-2029 > URL: https://issues.apache.org/jira/browse/HUDI-2029 > Project: Apache Hudi > Issue Type: Improvement > Components: Performance >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: pull-request-available > > Implement compression for DiskBasedMap in Spillable Map > Without compression, DiskBasedMap is causing more spilling to disk than > RockDb. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
codecov-commenter edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865384115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378381#comment-17378381 ] ASF GitHub Bot commented on HUDI-2029: -- codecov-commenter edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865384115 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3128](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2f8c083) into [master](https://codecov.io/gh/apache/hudi/commit/b4562e86e4b58d6151fdeea12e727b8c8881a213?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b4562e8) will **decrease** coverage by `20.23%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3128/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3128 +/- ## = - Coverage 47.63% 27.39% -20.24% + Complexity 5506 1287 -4219 = Files 930 381 -549 Lines 4127515115-26160 Branches 4138 1305 -2833 = - Hits 19661 4141-15520 + Misses1986710674 -9193 + Partials 1747 300 -1447 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.03% <0.00%> (-13.55%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.57% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-42.89%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_sourc
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
codecov-commenter edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865384115 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3128](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2f8c083) into [master](https://codecov.io/gh/apache/hudi/commit/b4562e86e4b58d6151fdeea12e727b8c8881a213?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b4562e8) will **decrease** coverage by `20.23%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3128/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3128 +/- ## = - Coverage 47.63% 27.39% -20.24% + Complexity 5506 1287 -4219 = Files 930 381 -549 Lines 4127515115-26160 Branches 4138 1305 -2833 = - Hits 19661 4141-15520 + Misses1986710674 -9193 + Partials 1747 300 -1447 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.03% <0.00%> (-13.55%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.57% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-42.89%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` |
[GitHub] [hudi] wangxianghu closed pull request #3248: [MINOR] Fix some wrong assert reasons
wangxianghu closed pull request #3248: URL: https://github.com/apache/hudi/pull/3248 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (HUDI-2147) Remove unused class AvroConvertor in hudi-flink
[ https://issues.apache.org/jira/browse/HUDI-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2147. -- Resolution: Done 3b2a4f2b6b49e13997292ecafa9accdd3e7b9efd > Remove unused class AvroConvertor in hudi-flink > --- > > Key: HUDI-2147 > URL: https://issues.apache.org/jira/browse/HUDI-2147 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2147) Remove unused class AvroConvertor in hudi-flink
[ https://issues.apache.org/jira/browse/HUDI-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2147: --- Fix Version/s: 0.9.0 > Remove unused class AvroConvertor in hudi-flink > --- > > Key: HUDI-2147 > URL: https://issues.apache.org/jira/browse/HUDI-2147 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2147) Remove unused class AvroConvertor in hudi-flink
[ https://issues.apache.org/jira/browse/HUDI-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378376#comment-17378376 ] ASF GitHub Bot commented on HUDI-2147: -- yanghua merged pull request #3243: URL: https://github.com/apache/hudi/pull/3243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove unused class AvroConvertor in hudi-flink > --- > > Key: HUDI-2147 > URL: https://issues.apache.org/jira/browse/HUDI-2147 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[hudi] branch master updated (b4562e8 -> 3b2a4f2)
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from b4562e8 Revert "[HUDI-2087] Support Append only in Flink stream (#3174)" (#3251) add 3b2a4f2 [HUDI-2147] Remove unused class AvroConvertor in hudi-flink (#3243) No new revisions were added by this update. Summary of changes: .../java/org/apache/hudi/util/AvroConvertor.java | 81 -- 1 file changed, 81 deletions(-) delete mode 100644 hudi-flink/src/main/java/org/apache/hudi/util/AvroConvertor.java
[GitHub] [hudi] yanghua merged pull request #3243: [HUDI-2147] Remove unused class AvroConvertor in hudi-flink
yanghua merged pull request #3243: URL: https://github.com/apache/hudi/pull/3243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378375#comment-17378375 ] ASF GitHub Bot commented on HUDI-2029: -- codecov-commenter edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865384115 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3128](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2f8c083) into [master](https://codecov.io/gh/apache/hudi/commit/b4562e86e4b58d6151fdeea12e727b8c8881a213?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b4562e8) will **decrease** coverage by `32.01%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3128/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3128 +/- ## = - Coverage 47.63% 15.61% -32.02% + Complexity 5506 489 -5017 = Files 930 281 -649 Lines 4127511627-29648 Branches 4138 952 -3186 = - Hits 19661 1816-17845 + Misses19867 9650-10217 + Partials 1747 161 -1586 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.59%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.57% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-42.89%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
codecov-commenter edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865384115 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3128](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2f8c083) into [master](https://codecov.io/gh/apache/hudi/commit/b4562e86e4b58d6151fdeea12e727b8c8881a213?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b4562e8) will **decrease** coverage by `32.01%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3128/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3128 +/- ## = - Coverage 47.63% 15.61% -32.02% + Complexity 5506 489 -5017 = Files 930 281 -649 Lines 4127511627-29648 Branches 4138 952 -3186 = - Hits 19661 1816-17845 + Misses19867 9650-10217 + Partials 1747 161 -1586 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.59%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.57% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-42.89%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` |
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378371#comment-17378371 ] ASF GitHub Bot commented on HUDI-2029: -- codecov-commenter edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865384115 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3128](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2f8c083) into [master](https://codecov.io/gh/apache/hudi/commit/b4562e86e4b58d6151fdeea12e727b8c8881a213?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b4562e8) will **decrease** coverage by `44.75%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3128/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3128 +/- ## - Coverage 47.63% 2.88% -44.76% + Complexity 5506 85 -5421 Files 930 281 -649 Lines 41275 11627-29648 Branches 4138 952 -3186 - Hits 19661 335-19326 + Misses19867 11266 -8601 + Partials 1747 26 -1721 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.59%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.25% <ø> (-49.36%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-42.89%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
codecov-commenter edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865384115 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3128](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2f8c083) into [master](https://codecov.io/gh/apache/hudi/commit/b4562e86e4b58d6151fdeea12e727b8c8881a213?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b4562e8) will **decrease** coverage by `44.75%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3128/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3128 +/- ## - Coverage 47.63% 2.88% -44.76% + Complexity 5506 85 -5421 Files 930 281 -649 Lines 41275 11627-29648 Branches 4138 952 -3186 - Hits 19661 335-19326 + Misses19867 11266 -8601 + Partials 1747 26 -1721 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.59%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.15%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.25% <ø> (-49.36%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3128?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-42.89%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3128/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sourc
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378366#comment-17378366 ] ASF GitHub Bot commented on HUDI-2029: -- hudi-bot edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865345707 ## CI report: * 2f8c0835c87a73417c3c0c05f1d515b61acba352 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=830) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement compression for DiskBasedMap in Spillable Map > --- > > Key: HUDI-2029 > URL: https://issues.apache.org/jira/browse/HUDI-2029 > Project: Apache Hudi > Issue Type: Improvement > Components: Performance >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: pull-request-available > > Implement compression for DiskBasedMap in Spillable Map > Without compression, DiskBasedMap is causing more spilling to disk than > RockDb. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
hudi-bot edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865345707 ## CI report: * 2f8c0835c87a73417c3c0c05f1d515b61acba352 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=830) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378364#comment-17378364 ] ASF GitHub Bot commented on HUDI-2029: -- hudi-bot edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865345707 ## CI report: * 973773c06c47af2ed5410c0938e690ea3aad7f48 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=330) * 2f8c0835c87a73417c3c0c05f1d515b61acba352 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=830) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement compression for DiskBasedMap in Spillable Map > --- > > Key: HUDI-2029 > URL: https://issues.apache.org/jira/browse/HUDI-2029 > Project: Apache Hudi > Issue Type: Improvement > Components: Performance >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: pull-request-available > > Implement compression for DiskBasedMap in Spillable Map > Without compression, DiskBasedMap is causing more spilling to disk than > RockDb. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
hudi-bot edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865345707 ## CI report: * 973773c06c47af2ed5410c0938e690ea3aad7f48 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=330) * 2f8c0835c87a73417c3c0c05f1d515b61acba352 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=830) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378363#comment-17378363 ] ASF GitHub Bot commented on HUDI-2029: -- hudi-bot edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865345707 ## CI report: * 973773c06c47af2ed5410c0938e690ea3aad7f48 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=330) * 2f8c0835c87a73417c3c0c05f1d515b61acba352 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement compression for DiskBasedMap in Spillable Map > --- > > Key: HUDI-2029 > URL: https://issues.apache.org/jira/browse/HUDI-2029 > Project: Apache Hudi > Issue Type: Improvement > Components: Performance >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: pull-request-available > > Implement compression for DiskBasedMap in Spillable Map > Without compression, DiskBasedMap is causing more spilling to disk than > RockDb. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378362#comment-17378362 ] ASF GitHub Bot commented on HUDI-2029: -- rmahindra123 commented on a change in pull request #3128: URL: https://github.com/apache/hudi/pull/3128#discussion_r667259394 ## File path: hudi-common/src/test/java/org/apache/hudi/common/util/collection/TestBitCaskDiskMap.java ## @@ -66,27 +68,33 @@ public void setup() { initPath(); } - @Test - public void testSimpleInsert() throws IOException, URISyntaxException { -BitCaskDiskMap records = new BitCaskDiskMap<>(basePath); + @ParameterizedTest + @ValueSource(booleans = {false, true}) + public void testSimpleInsert(boolean isCompressionEnabled) throws IOException, URISyntaxException { +BitCaskDiskMap records = new BitCaskDiskMap<>(basePath, isCompressionEnabled); List iRecords = SchemaTestUtil.generateHoodieTestRecords(0, 100); -((GenericRecord) iRecords.get(0)).get(HoodieRecord.COMMIT_TIME_METADATA_FIELD).toString(); List recordKeys = SpillableMapTestUtils.upsertRecords(iRecords, records); +Map originalRecords = iRecords.stream() +.collect(Collectors.toMap(k -> ((GenericRecord) k).get(HoodieRecord.RECORD_KEY_METADATA_FIELD).toString(), v -> v)); + // make sure records have spilled to disk assertTrue(records.sizeOfFileOnDiskInBytes() > 0); Iterator> itr = records.iterator(); -List oRecords = new ArrayList<>(); while (itr.hasNext()) { HoodieRecord rec = itr.next(); - oRecords.add(rec); assert recordKeys.contains(rec.getRecordKey()); + IndexedRecord originalRecord = originalRecords.get(rec.getRecordKey()); + HoodieAvroPayload payload = (HoodieAvroPayload) rec.getData(); + Option value = payload.getInsertValue(HoodieAvroUtils.addMetadataFields(getSimpleSchema())); + assertEquals(originalRecord, value.get()); Review comment: Added testing for value, that should test compression/decompression -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement compression for DiskBasedMap in Spillable Map > --- > > Key: HUDI-2029 > URL: https://issues.apache.org/jira/browse/HUDI-2029 > Project: Apache Hudi > Issue Type: Improvement > Components: Performance >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: pull-request-available > > Implement compression for DiskBasedMap in Spillable Map > Without compression, DiskBasedMap is causing more spilling to disk than > RockDb. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
hudi-bot edited a comment on pull request #3128: URL: https://github.com/apache/hudi/pull/3128#issuecomment-865345707 ## CI report: * 973773c06c47af2ed5410c0938e690ea3aad7f48 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=330) * 2f8c0835c87a73417c3c0c05f1d515b61acba352 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] rmahindra123 commented on a change in pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
rmahindra123 commented on a change in pull request #3128: URL: https://github.com/apache/hudi/pull/3128#discussion_r667259394 ## File path: hudi-common/src/test/java/org/apache/hudi/common/util/collection/TestBitCaskDiskMap.java ## @@ -66,27 +68,33 @@ public void setup() { initPath(); } - @Test - public void testSimpleInsert() throws IOException, URISyntaxException { -BitCaskDiskMap records = new BitCaskDiskMap<>(basePath); + @ParameterizedTest + @ValueSource(booleans = {false, true}) + public void testSimpleInsert(boolean isCompressionEnabled) throws IOException, URISyntaxException { +BitCaskDiskMap records = new BitCaskDiskMap<>(basePath, isCompressionEnabled); List iRecords = SchemaTestUtil.generateHoodieTestRecords(0, 100); -((GenericRecord) iRecords.get(0)).get(HoodieRecord.COMMIT_TIME_METADATA_FIELD).toString(); List recordKeys = SpillableMapTestUtils.upsertRecords(iRecords, records); +Map originalRecords = iRecords.stream() +.collect(Collectors.toMap(k -> ((GenericRecord) k).get(HoodieRecord.RECORD_KEY_METADATA_FIELD).toString(), v -> v)); + // make sure records have spilled to disk assertTrue(records.sizeOfFileOnDiskInBytes() > 0); Iterator> itr = records.iterator(); -List oRecords = new ArrayList<>(); while (itr.hasNext()) { HoodieRecord rec = itr.next(); - oRecords.add(rec); assert recordKeys.contains(rec.getRecordKey()); + IndexedRecord originalRecord = originalRecords.get(rec.getRecordKey()); + HoodieAvroPayload payload = (HoodieAvroPayload) rec.getData(); + Option value = payload.getInsertValue(HoodieAvroUtils.addMetadataFields(getSimpleSchema())); + assertEquals(originalRecord, value.get()); Review comment: Added testing for value, that should test compression/decompression -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] rmahindra123 commented on a change in pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
rmahindra123 commented on a change in pull request #3128: URL: https://github.com/apache/hudi/pull/3128#discussion_r667255510 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java ## @@ -395,4 +417,48 @@ public int compareTo(ValueMetadata o) { return Long.compare(this.offsetOfValue, o.offsetOfValue); } } + + private static class DiskCompressionInstance implements Serializable { +public static final int DISK_COMPRESSION_INITIAL_BUFFER_SIZE = 1048576; + +// Caching ByteArrayOutputStreams to avoid recreating it for every operation +private final ByteArrayOutputStream compressBaos; +private final ByteArrayOutputStream decompressBaos; +private final byte[] decompressBuffer; + +DiskCompressionInstance() { + compressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBuffer = new byte[8192]; +} + +public byte[] compressBytes(final byte [] value) throws IOException { + compressBaos.reset(); Review comment: sorry did not follow here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378357#comment-17378357 ] ASF GitHub Bot commented on HUDI-2029: -- rmahindra123 commented on a change in pull request #3128: URL: https://github.com/apache/hudi/pull/3128#discussion_r667255510 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java ## @@ -395,4 +417,48 @@ public int compareTo(ValueMetadata o) { return Long.compare(this.offsetOfValue, o.offsetOfValue); } } + + private static class DiskCompressionInstance implements Serializable { +public static final int DISK_COMPRESSION_INITIAL_BUFFER_SIZE = 1048576; + +// Caching ByteArrayOutputStreams to avoid recreating it for every operation +private final ByteArrayOutputStream compressBaos; +private final ByteArrayOutputStream decompressBaos; +private final byte[] decompressBuffer; + +DiskCompressionInstance() { + compressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBuffer = new byte[8192]; +} + +public byte[] compressBytes(final byte [] value) throws IOException { + compressBaos.reset(); Review comment: sorry did not follow here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement compression for DiskBasedMap in Spillable Map > --- > > Key: HUDI-2029 > URL: https://issues.apache.org/jira/browse/HUDI-2029 > Project: Apache Hudi > Issue Type: Improvement > Components: Performance >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: pull-request-available > > Implement compression for DiskBasedMap in Spillable Map > Without compression, DiskBasedMap is causing more spilling to disk than > RockDb. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378356#comment-17378356 ] ASF GitHub Bot commented on HUDI-2029: -- rmahindra123 commented on a change in pull request #3128: URL: https://github.com/apache/hudi/pull/3128#discussion_r667254655 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java ## @@ -395,4 +417,48 @@ public int compareTo(ValueMetadata o) { return Long.compare(this.offsetOfValue, o.offsetOfValue); } } + + private static class DiskCompressionInstance implements Serializable { +public static final int DISK_COMPRESSION_INITIAL_BUFFER_SIZE = 1048576; + +// Caching ByteArrayOutputStreams to avoid recreating it for every operation +private final ByteArrayOutputStream compressBaos; +private final ByteArrayOutputStream decompressBaos; +private final byte[] decompressBuffer; + +DiskCompressionInstance() { + compressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBuffer = new byte[8192]; +} + +public byte[] compressBytes(final byte [] value) throws IOException { Review comment: Yeah this is optimized for running as a single thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement compression for DiskBasedMap in Spillable Map > --- > > Key: HUDI-2029 > URL: https://issues.apache.org/jira/browse/HUDI-2029 > Project: Apache Hudi > Issue Type: Improvement > Components: Performance >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: pull-request-available > > Implement compression for DiskBasedMap in Spillable Map > Without compression, DiskBasedMap is causing more spilling to disk than > RockDb. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2029) Implement compression for DiskBasedMap in Spillable Map
[ https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378355#comment-17378355 ] ASF GitHub Bot commented on HUDI-2029: -- rmahindra123 commented on a change in pull request #3128: URL: https://github.com/apache/hudi/pull/3128#discussion_r667254510 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java ## @@ -395,4 +417,48 @@ public int compareTo(ValueMetadata o) { return Long.compare(this.offsetOfValue, o.offsetOfValue); } } + + private static class DiskCompressionInstance implements Serializable { +public static final int DISK_COMPRESSION_INITIAL_BUFFER_SIZE = 1048576; + +// Caching ByteArrayOutputStreams to avoid recreating it for every operation +private final ByteArrayOutputStream compressBaos; +private final ByteArrayOutputStream decompressBaos; +private final byte[] decompressBuffer; + +DiskCompressionInstance() { + compressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBuffer = new byte[8192]; Review comment: Yes, it is dynamic and uses ByteArrayOutputStream (decompressBaos). decompressBuffer is just a intermediate buffer that reads from the inputStream and writes to the outputstream, aka decompressBaos. So 8192 is the max bytes read at a given time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement compression for DiskBasedMap in Spillable Map > --- > > Key: HUDI-2029 > URL: https://issues.apache.org/jira/browse/HUDI-2029 > Project: Apache Hudi > Issue Type: Improvement > Components: Performance >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: pull-request-available > > Implement compression for DiskBasedMap in Spillable Map > Without compression, DiskBasedMap is causing more spilling to disk than > RockDb. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] rmahindra123 commented on a change in pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
rmahindra123 commented on a change in pull request #3128: URL: https://github.com/apache/hudi/pull/3128#discussion_r667254655 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java ## @@ -395,4 +417,48 @@ public int compareTo(ValueMetadata o) { return Long.compare(this.offsetOfValue, o.offsetOfValue); } } + + private static class DiskCompressionInstance implements Serializable { +public static final int DISK_COMPRESSION_INITIAL_BUFFER_SIZE = 1048576; + +// Caching ByteArrayOutputStreams to avoid recreating it for every operation +private final ByteArrayOutputStream compressBaos; +private final ByteArrayOutputStream decompressBaos; +private final byte[] decompressBuffer; + +DiskCompressionInstance() { + compressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBuffer = new byte[8192]; +} + +public byte[] compressBytes(final byte [] value) throws IOException { Review comment: Yeah this is optimized for running as a single thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] rmahindra123 commented on a change in pull request #3128: [HUDI-2029] Implement compression for DiskBasedMap in Spillable Map
rmahindra123 commented on a change in pull request #3128: URL: https://github.com/apache/hudi/pull/3128#discussion_r667254510 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java ## @@ -395,4 +417,48 @@ public int compareTo(ValueMetadata o) { return Long.compare(this.offsetOfValue, o.offsetOfValue); } } + + private static class DiskCompressionInstance implements Serializable { +public static final int DISK_COMPRESSION_INITIAL_BUFFER_SIZE = 1048576; + +// Caching ByteArrayOutputStreams to avoid recreating it for every operation +private final ByteArrayOutputStream compressBaos; +private final ByteArrayOutputStream decompressBaos; +private final byte[] decompressBuffer; + +DiskCompressionInstance() { + compressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBaos = new ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE); + decompressBuffer = new byte[8192]; Review comment: Yes, it is dynamic and uses ByteArrayOutputStream (decompressBaos). decompressBuffer is just a intermediate buffer that reads from the inputStream and writes to the outputstream, aka decompressBaos. So 8192 is the max bytes read at a given time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378354#comment-17378354 ] ASF GitHub Bot commented on HUDI-1860: -- codecov-commenter edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870526141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Samrat Deb >Priority: Major > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
codecov-commenter edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870526141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378352#comment-17378352 ] ASF GitHub Bot commented on HUDI-2087: -- codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (701c28f) into [master](https://codecov.io/gh/apache/hudi/commit/221ddd9bf3899e3672210404d51e686770ba446d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (221ddd9) will **decrease** coverage by `18.44%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## = - Coverage 45.85% 27.40% -18.45% + Complexity 4699 1287 -3412 = Files 828 381 -447 Lines 3774215108-22634 Branches 3780 1305 -2475 = - Hits 17305 4141-13164 + Misses1883310667 -8166 + Partials 1604 300 -1304 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.05% <0.00%> (-1.93%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.11%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.57% <ø> (-0.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referr
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (701c28f) into [master](https://codecov.io/gh/apache/hudi/commit/221ddd9bf3899e3672210404d51e686770ba446d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (221ddd9) will **decrease** coverage by `18.44%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## = - Coverage 45.85% 27.40% -18.45% + Complexity 4699 1287 -3412 = Files 828 381 -447 Lines 3774215108-22634 Branches 3780 1305 -2475 = - Hits 17305 4141-13164 + Misses1883310667 -8166 + Partials 1604 300 -1304 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.05% <0.00%> (-1.93%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.11%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.57% <ø> (-0.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378344#comment-17378344 ] ASF GitHub Bot commented on HUDI-2087: -- codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (701c28f) into [master](https://codecov.io/gh/apache/hudi/commit/221ddd9bf3899e3672210404d51e686770ba446d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (221ddd9) will **decrease** coverage by `18.44%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## = - Coverage 45.85% 27.40% -18.45% + Complexity 4699 1287 -3412 = Files 828 381 -447 Lines 3774215108-22634 Branches 3780 1305 -2475 = - Hits 17305 4141-13164 + Misses1883310667 -8166 + Partials 1604 300 -1304 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.05% <0.00%> (-1.93%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.11%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.57% <ø> (-0.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referr
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3252: [HUDI-2087] Support Append only in Flink stream
codecov-commenter edited a comment on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (701c28f) into [master](https://codecov.io/gh/apache/hudi/commit/221ddd9bf3899e3672210404d51e686770ba446d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (221ddd9) will **decrease** coverage by `18.44%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## = - Coverage 45.85% 27.40% -18.45% + Complexity 4699 1287 -3412 = Files 828 381 -447 Lines 3774215108-22634 Branches 3780 1305 -2475 = - Hits 17305 4141-13164 + Misses1883310667 -8166 + Partials 1604 300 -1304 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.05% <0.00%> (-1.93%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.11%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.57% <ø> (-0.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00
[jira] [Comment Edited] (HUDI-2159) Supporting Clustering and Metadata Table together
[ https://issues.apache.org/jira/browse/HUDI-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378342#comment-17378342 ] Vinoth Chandar edited comment on HUDI-2159 at 7/9/21, 10:52 PM: >Metadata Table sync only works in completion order. I almost feels like, this is the sticking point in all the issues we hit :) . We gained debuggability with the sync stuff. but there is too much complexity we incurred in other ways? was (Author: vc): >Metadata Table sync only works in completion order. I almost feels like, this is the sticking point in all the issues we hit :) . We gained debuggability with the sync stuff. but there is too much complexity. > Supporting Clustering and Metadata Table together > - > > Key: HUDI-2159 > URL: https://issues.apache.org/jira/browse/HUDI-2159 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Prashant Wason >Assignee: Prashant Wason >Priority: Blocker > Fix For: 0.9.0 > > > I am testing clustering support for metadata enabled table and found a few > issues. > *Setup* > Pipeline 1: Ingestion pipeline with Metadata Table enabled. Runs every 30 > mins. > Pipeline 2: Clustering pipeline with long running jobs (3-4 hours) > Pipeline 3: Another clustering pipeline with long running jobs (3-4 hours) > > *Issue #1: Parallel commits on Metadata Table* > Assume the Clustering pipeline is completing T5.replacecommit and ingestion > pipeline is completing T10.commit. Metadata Table will synced at an instant > Now both the pipelines will call syncMetadataTable() which will do the > following: > # Find all un-synced instants from dataset (T5, T6 ... T10) > # Read each instant and perform a deltacommit on the Metadata Table with the > same timestamp as instant. > There is a chance that two processed perform deltacommit at T5 on the > metadata table and one will fail (instant file already exists). This will be > an exception raised and will be detected as failure of pipeline leading to > false-positive alerts. > > *Issue #2: No archiving/rollback support for failed clustering operations* > If a clustering operation fails, it leaves a left-over > T5.replacecommit.inflight. There is no automated way to rollback or archive > these. Since clustering is a long running operation in general and may be run > through multiple pipelines at the same time, automated rollback of left-over > inflights doesnt work as we cannot be sure that the process is dead. > Metadata Table sync only works in completion order. So if > T5.replacecommit.inflight is left-over, Metadata Table will not sync beyond > T5 causing a large number of LogBLocks to pile up which will have to be > merged in memory leading to deteriorating performance. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2159) Supporting Clustering and Metadata Table together
[ https://issues.apache.org/jira/browse/HUDI-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378342#comment-17378342 ] Vinoth Chandar commented on HUDI-2159: -- >Metadata Table sync only works in completion order. I almost feels like, this is the sticking point in all the issues we hit :) . We gained debuggability with the sync stuff. but there is too much complexity. > Supporting Clustering and Metadata Table together > - > > Key: HUDI-2159 > URL: https://issues.apache.org/jira/browse/HUDI-2159 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Prashant Wason >Assignee: Prashant Wason >Priority: Blocker > Fix For: 0.9.0 > > > I am testing clustering support for metadata enabled table and found a few > issues. > *Setup* > Pipeline 1: Ingestion pipeline with Metadata Table enabled. Runs every 30 > mins. > Pipeline 2: Clustering pipeline with long running jobs (3-4 hours) > Pipeline 3: Another clustering pipeline with long running jobs (3-4 hours) > > *Issue #1: Parallel commits on Metadata Table* > Assume the Clustering pipeline is completing T5.replacecommit and ingestion > pipeline is completing T10.commit. Metadata Table will synced at an instant > Now both the pipelines will call syncMetadataTable() which will do the > following: > # Find all un-synced instants from dataset (T5, T6 ... T10) > # Read each instant and perform a deltacommit on the Metadata Table with the > same timestamp as instant. > There is a chance that two processed perform deltacommit at T5 on the > metadata table and one will fail (instant file already exists). This will be > an exception raised and will be detected as failure of pipeline leading to > false-positive alerts. > > *Issue #2: No archiving/rollback support for failed clustering operations* > If a clustering operation fails, it leaves a left-over > T5.replacecommit.inflight. There is no automated way to rollback or archive > these. Since clustering is a long running operation in general and may be run > through multiple pipelines at the same time, automated rollback of left-over > inflights doesnt work as we cannot be sure that the process is dead. > Metadata Table sync only works in completion order. So if > T5.replacecommit.inflight is left-over, Metadata Table will not sync beyond > T5 causing a large number of LogBLocks to pile up which will have to be > merged in memory leading to deteriorating performance. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HUDI-2159) Supporting Clustering and Metadata Table together
[ https://issues.apache.org/jira/browse/HUDI-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378341#comment-17378341 ] Vinoth Chandar edited comment on HUDI-2159 at 7/9/21, 10:49 PM: > Since, ingestion runs at faster cadence, we can set hoodie.metadata.sync=true >in ingestion pipeline as hoodie.metadata.sync=false in all other pipelines. This is a practical approach. I wonder again though, if the multi writer stuff already have something like this. I feel 2 is complex. was (Author: vc): > Since, ingestion runs at faster cadence, we can set hoodie.metadata.sync=true >in ingestion pipeline as hoodie.metadata.sync=false in all other pipelines. This is a practical approach. I wonder again though, if the multi writer stuff already have something like this. > Supporting Clustering and Metadata Table together > - > > Key: HUDI-2159 > URL: https://issues.apache.org/jira/browse/HUDI-2159 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Prashant Wason >Assignee: Prashant Wason >Priority: Blocker > Fix For: 0.9.0 > > > I am testing clustering support for metadata enabled table and found a few > issues. > *Setup* > Pipeline 1: Ingestion pipeline with Metadata Table enabled. Runs every 30 > mins. > Pipeline 2: Clustering pipeline with long running jobs (3-4 hours) > Pipeline 3: Another clustering pipeline with long running jobs (3-4 hours) > > *Issue #1: Parallel commits on Metadata Table* > Assume the Clustering pipeline is completing T5.replacecommit and ingestion > pipeline is completing T10.commit. Metadata Table will synced at an instant > Now both the pipelines will call syncMetadataTable() which will do the > following: > # Find all un-synced instants from dataset (T5, T6 ... T10) > # Read each instant and perform a deltacommit on the Metadata Table with the > same timestamp as instant. > There is a chance that two processed perform deltacommit at T5 on the > metadata table and one will fail (instant file already exists). This will be > an exception raised and will be detected as failure of pipeline leading to > false-positive alerts. > > *Issue #2: No archiving/rollback support for failed clustering operations* > If a clustering operation fails, it leaves a left-over > T5.replacecommit.inflight. There is no automated way to rollback or archive > these. Since clustering is a long running operation in general and may be run > through multiple pipelines at the same time, automated rollback of left-over > inflights doesnt work as we cannot be sure that the process is dead. > Metadata Table sync only works in completion order. So if > T5.replacecommit.inflight is left-over, Metadata Table will not sync beyond > T5 causing a large number of LogBLocks to pile up which will have to be > merged in memory leading to deteriorating performance. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2159) Supporting Clustering and Metadata Table together
[ https://issues.apache.org/jira/browse/HUDI-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378341#comment-17378341 ] Vinoth Chandar commented on HUDI-2159: -- > Since, ingestion runs at faster cadence, we can set hoodie.metadata.sync=true >in ingestion pipeline as hoodie.metadata.sync=false in all other pipelines. This is a practical approach. I wonder again though, if the multi writer stuff already have something like this. > Supporting Clustering and Metadata Table together > - > > Key: HUDI-2159 > URL: https://issues.apache.org/jira/browse/HUDI-2159 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Prashant Wason >Assignee: Prashant Wason >Priority: Blocker > Fix For: 0.9.0 > > > I am testing clustering support for metadata enabled table and found a few > issues. > *Setup* > Pipeline 1: Ingestion pipeline with Metadata Table enabled. Runs every 30 > mins. > Pipeline 2: Clustering pipeline with long running jobs (3-4 hours) > Pipeline 3: Another clustering pipeline with long running jobs (3-4 hours) > > *Issue #1: Parallel commits on Metadata Table* > Assume the Clustering pipeline is completing T5.replacecommit and ingestion > pipeline is completing T10.commit. Metadata Table will synced at an instant > Now both the pipelines will call syncMetadataTable() which will do the > following: > # Find all un-synced instants from dataset (T5, T6 ... T10) > # Read each instant and perform a deltacommit on the Metadata Table with the > same timestamp as instant. > There is a chance that two processed perform deltacommit at T5 on the > metadata table and one will fail (instant file already exists). This will be > an exception raised and will be detected as failure of pipeline leading to > false-positive alerts. > > *Issue #2: No archiving/rollback support for failed clustering operations* > If a clustering operation fails, it leaves a left-over > T5.replacecommit.inflight. There is no automated way to rollback or archive > these. Since clustering is a long running operation in general and may be run > through multiple pipelines at the same time, automated rollback of left-over > inflights doesnt work as we cannot be sure that the process is dead. > Metadata Table sync only works in completion order. So if > T5.replacecommit.inflight is left-over, Metadata Table will not sync beyond > T5 causing a large number of LogBLocks to pile up which will have to be > merged in memory leading to deteriorating performance. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2159) Supporting Clustering and Metadata Table together
[ https://issues.apache.org/jira/browse/HUDI-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378340#comment-17378340 ] Vinoth Chandar commented on HUDI-2159: -- >There is a chance that two processed perform deltacommit at T5 on the metadata >table and one will fail (instant file already exists). Would n't the locking service we do for multi writer solve all this? > Supporting Clustering and Metadata Table together > - > > Key: HUDI-2159 > URL: https://issues.apache.org/jira/browse/HUDI-2159 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Prashant Wason >Assignee: Prashant Wason >Priority: Blocker > Fix For: 0.9.0 > > > I am testing clustering support for metadata enabled table and found a few > issues. > *Setup* > Pipeline 1: Ingestion pipeline with Metadata Table enabled. Runs every 30 > mins. > Pipeline 2: Clustering pipeline with long running jobs (3-4 hours) > Pipeline 3: Another clustering pipeline with long running jobs (3-4 hours) > > *Issue #1: Parallel commits on Metadata Table* > Assume the Clustering pipeline is completing T5.replacecommit and ingestion > pipeline is completing T10.commit. Metadata Table will synced at an instant > Now both the pipelines will call syncMetadataTable() which will do the > following: > # Find all un-synced instants from dataset (T5, T6 ... T10) > # Read each instant and perform a deltacommit on the Metadata Table with the > same timestamp as instant. > There is a chance that two processed perform deltacommit at T5 on the > metadata table and one will fail (instant file already exists). This will be > an exception raised and will be detected as failure of pipeline leading to > false-positive alerts. > > *Issue #2: No archiving/rollback support for failed clustering operations* > If a clustering operation fails, it leaves a left-over > T5.replacecommit.inflight. There is no automated way to rollback or archive > these. Since clustering is a long running operation in general and may be run > through multiple pipelines at the same time, automated rollback of left-over > inflights doesnt work as we cannot be sure that the process is dead. > Metadata Table sync only works in completion order. So if > T5.replacecommit.inflight is left-over, Metadata Table will not sync beyond > T5 causing a large number of LogBLocks to pile up which will have to be > merged in memory leading to deteriorating performance. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2159) Supporting Clustering and Metadata Table together
[ https://issues.apache.org/jira/browse/HUDI-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2159: - Fix Version/s: 0.9.0 > Supporting Clustering and Metadata Table together > - > > Key: HUDI-2159 > URL: https://issues.apache.org/jira/browse/HUDI-2159 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Prashant Wason >Assignee: Prashant Wason >Priority: Blocker > Fix For: 0.9.0 > > > I am testing clustering support for metadata enabled table and found a few > issues. > *Setup* > Pipeline 1: Ingestion pipeline with Metadata Table enabled. Runs every 30 > mins. > Pipeline 2: Clustering pipeline with long running jobs (3-4 hours) > Pipeline 3: Another clustering pipeline with long running jobs (3-4 hours) > > *Issue #1: Parallel commits on Metadata Table* > Assume the Clustering pipeline is completing T5.replacecommit and ingestion > pipeline is completing T10.commit. Metadata Table will synced at an instant > Now both the pipelines will call syncMetadataTable() which will do the > following: > # Find all un-synced instants from dataset (T5, T6 ... T10) > # Read each instant and perform a deltacommit on the Metadata Table with the > same timestamp as instant. > There is a chance that two processed perform deltacommit at T5 on the > metadata table and one will fail (instant file already exists). This will be > an exception raised and will be detected as failure of pipeline leading to > false-positive alerts. > > *Issue #2: No archiving/rollback support for failed clustering operations* > If a clustering operation fails, it leaves a left-over > T5.replacecommit.inflight. There is no automated way to rollback or archive > these. Since clustering is a long running operation in general and may be run > through multiple pipelines at the same time, automated rollback of left-over > inflights doesnt work as we cannot be sure that the process is dead. > Metadata Table sync only works in completion order. So if > T5.replacecommit.inflight is left-over, Metadata Table will not sync beyond > T5 causing a large number of LogBLocks to pile up which will have to be > merged in memory leading to deteriorating performance. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2159) Supporting Clustering and Metadata Table together
[ https://issues.apache.org/jira/browse/HUDI-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2159: - Priority: Blocker (was: Major) > Supporting Clustering and Metadata Table together > - > > Key: HUDI-2159 > URL: https://issues.apache.org/jira/browse/HUDI-2159 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Prashant Wason >Assignee: Prashant Wason >Priority: Blocker > > I am testing clustering support for metadata enabled table and found a few > issues. > *Setup* > Pipeline 1: Ingestion pipeline with Metadata Table enabled. Runs every 30 > mins. > Pipeline 2: Clustering pipeline with long running jobs (3-4 hours) > Pipeline 3: Another clustering pipeline with long running jobs (3-4 hours) > > *Issue #1: Parallel commits on Metadata Table* > Assume the Clustering pipeline is completing T5.replacecommit and ingestion > pipeline is completing T10.commit. Metadata Table will synced at an instant > Now both the pipelines will call syncMetadataTable() which will do the > following: > # Find all un-synced instants from dataset (T5, T6 ... T10) > # Read each instant and perform a deltacommit on the Metadata Table with the > same timestamp as instant. > There is a chance that two processed perform deltacommit at T5 on the > metadata table and one will fail (instant file already exists). This will be > an exception raised and will be detected as failure of pipeline leading to > false-positive alerts. > > *Issue #2: No archiving/rollback support for failed clustering operations* > If a clustering operation fails, it leaves a left-over > T5.replacecommit.inflight. There is no automated way to rollback or archive > these. Since clustering is a long running operation in general and may be run > through multiple pipelines at the same time, automated rollback of left-over > inflights doesnt work as we cannot be sure that the process is dead. > Metadata Table sync only works in completion order. So if > T5.replacecommit.inflight is left-over, Metadata Table will not sync beyond > T5 causing a large number of LogBLocks to pile up which will have to be > merged in memory leading to deteriorating performance. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378339#comment-17378339 ] ASF GitHub Bot commented on HUDI-1860: -- codecov-commenter edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870526141 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3184](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d63d6c5) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **decrease** coverage by `24.52%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3184/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3184 +/- ## - Coverage 27.40% 2.88% -24.53% + Complexity 1287 85 -1202 Files 381 281 -100 Lines 15108 11622 -3486 Branches 1305 952 -353 - Hits 4141 335 -3806 - Misses10667 11261 +594 + Partials300 26 -274 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <ø> (-21.06%)` | :arrow_down: | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `9.25% <0.00%> (-49.33%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-71.15%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
codecov-commenter edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870526141 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3184](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d63d6c5) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **decrease** coverage by `24.52%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3184/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3184 +/- ## - Coverage 27.40% 2.88% -24.53% + Complexity 1287 85 -1202 Files 381 281 -100 Lines 15108 11622 -3486 Branches 1305 952 -353 - Hits 4141 335 -3806 - Misses10667 11261 +594 + Partials300 26 -274 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <ø> (-21.06%)` | :arrow_down: | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `9.25% <0.00%> (-49.33%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-71.15%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Soft
[jira] [Commented] (HUDI-2087) Support Append only in Flink stream
[ https://issues.apache.org/jira/browse/HUDI-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378337#comment-17378337 ] ASF GitHub Bot commented on HUDI-2087: -- codecov-commenter commented on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (701c28f) into [master](https://codecov.io/gh/apache/hudi/commit/221ddd9bf3899e3672210404d51e686770ba446d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (221ddd9) will **decrease** coverage by `18.44%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## = - Coverage 45.85% 27.40% -18.45% + Complexity 4699 1287 -3412 = Files 828 381 -447 Lines 3774215108-22634 Branches 3780 1305 -2475 = - Hits 17305 4141-13164 + Misses1883310667 -8166 + Partials 1604 300 -1304 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.05% <0.00%> (-1.93%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.11%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.57% <ø> (-0.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_
[GitHub] [hudi] codecov-commenter commented on pull request #3252: [HUDI-2087] Support Append only in Flink stream
codecov-commenter commented on pull request #3252: URL: https://github.com/apache/hudi/pull/3252#issuecomment-877492128 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3252](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (701c28f) into [master](https://codecov.io/gh/apache/hudi/commit/221ddd9bf3899e3672210404d51e686770ba446d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (221ddd9) will **decrease** coverage by `18.44%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3252/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3252 +/- ## = - Coverage 45.85% 27.40% -18.45% + Complexity 4699 1287 -3412 = Files 828 381 -447 Lines 3774215108-22634 Branches 3780 1305 -2475 = - Hits 17305 4141-13164 + Misses1883310667 -8166 + Partials 1604 300 -1304 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.05% <0.00%> (-1.93%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.37% <ø> (-49.11%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.57% <ø> (-0.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/action/commit/BucketType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21taXQvQnVja2V0VHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3252/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-10
[jira] [Commented] (HUDI-1483) async clustering for deltastreamer
[ https://issues.apache.org/jira/browse/HUDI-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378323#comment-17378323 ] ASF GitHub Bot commented on HUDI-1483: -- codecov-commenter edited a comment on pull request #3142: URL: https://github.com/apache/hudi/pull/3142#issuecomment-867078369 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3142](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f86f50e) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **increase** coverage by `0.12%`. > The diff coverage is `34.84%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3142/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3142 +/- ## + Coverage 27.40% 27.53% +0.12% - Complexity 1287 1291 +4 Files 381 385 +4 Lines 1510815214 +106 Branches 1305 1316 +11 + Hits 4141 4189 +48 - Misses1066710722 +55 - Partials300 303 +3 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `20.93% <0.00%> (-0.12%)` | :arrow_down: | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `59.26% <90.19%> (+0.69%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/async/AsyncClusteringService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/async/HoodieAsyncService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/client/AbstractClusteringClient.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdENsdXN0ZXJpbmdDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/config/HoodieClusteringConfig.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVDbHVzdGVyaW5nQ29uZmlnLmphdmE=)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3142: [HUDI-1483] Support async clustering for deltastreamer and Spark streaming
codecov-commenter edited a comment on pull request #3142: URL: https://github.com/apache/hudi/pull/3142#issuecomment-867078369 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3142](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f86f50e) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **increase** coverage by `0.12%`. > The diff coverage is `34.84%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3142/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3142 +/- ## + Coverage 27.40% 27.53% +0.12% - Complexity 1287 1291 +4 Files 381 385 +4 Lines 1510815214 +106 Branches 1305 1316 +11 + Hits 4141 4189 +48 - Misses1066710722 +55 - Partials300 303 +3 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `20.93% <0.00%> (-0.12%)` | :arrow_down: | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `59.26% <90.19%> (+0.69%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/async/AsyncClusteringService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/async/HoodieAsyncService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/client/AbstractClusteringClient.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdENsdXN0ZXJpbmdDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/config/HoodieClusteringConfig.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVDbHVzdGVyaW5nQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apa
[jira] [Commented] (HUDI-1483) async clustering for deltastreamer
[ https://issues.apache.org/jira/browse/HUDI-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378321#comment-17378321 ] ASF GitHub Bot commented on HUDI-1483: -- codecov-commenter edited a comment on pull request #3142: URL: https://github.com/apache/hudi/pull/3142#issuecomment-867078369 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3142](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f86f50e) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **decrease** coverage by `11.49%`. > The diff coverage is `40.35%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3142/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3142 +/- ## = - Coverage 27.40% 15.91% -11.50% + Complexity 1287 493 -794 = Files 381 283 -98 Lines 1510811710 -3398 Branches 1305 961 -344 = - Hits 4141 1864 -2277 + Misses10667 9683 -984 + Partials300 163 -137 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <0.00%> (-21.06%)` | :arrow_down: | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `59.26% <90.19%> (+0.69%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/async/AsyncClusteringService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/async/HoodieAsyncService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/client/AbstractClusteringClient.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdENsdXN0ZXJpbmdDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/config/HoodieClusteringConfig.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVDbHVzdGVyaW5nQ29u
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3142: [HUDI-1483] Support async clustering for deltastreamer and Spark streaming
codecov-commenter edited a comment on pull request #3142: URL: https://github.com/apache/hudi/pull/3142#issuecomment-867078369 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3142](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f86f50e) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **decrease** coverage by `11.49%`. > The diff coverage is `40.35%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3142/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3142 +/- ## = - Coverage 27.40% 15.91% -11.50% + Complexity 1287 493 -794 = Files 381 283 -98 Lines 1510811710 -3398 Branches 1305 961 -344 = - Hits 4141 1864 -2277 + Misses10667 9683 -984 + Partials300 163 -137 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <0.00%> (-21.06%)` | :arrow_down: | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `59.26% <90.19%> (+0.69%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/async/AsyncClusteringService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/async/HoodieAsyncService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/client/AbstractClusteringClient.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdENsdXN0ZXJpbmdDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/config/HoodieClusteringConfig.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVDbHVzdGVyaW5nQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&ut
[jira] [Commented] (HUDI-1483) async clustering for deltastreamer
[ https://issues.apache.org/jira/browse/HUDI-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378318#comment-17378318 ] ASF GitHub Bot commented on HUDI-1483: -- codecov-commenter edited a comment on pull request #3142: URL: https://github.com/apache/hudi/pull/3142#issuecomment-867078369 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3142](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f86f50e) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **decrease** coverage by `24.54%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3142/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3142 +/- ## - Coverage 27.40% 2.86% -24.55% + Complexity 1287 85 -1202 Files 381 283 -98 Lines 15108 11710 -3398 Branches 1305 961 -344 - Hits 4141 335 -3806 - Misses10667 11349 +682 + Partials300 26 -274 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <0.00%> (-21.06%)` | :arrow_down: | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `9.11% <0.00%> (-49.46%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/async/AsyncClusteringService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/async/HoodieAsyncService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/client/AbstractClusteringClient.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdENsdXN0ZXJpbmdDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/config/HoodieClusteringConfig.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVDbHVzdGVyaW5nQ29uZmlnLmphdmE=)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3142: [HUDI-1483] Support async clustering for deltastreamer and Spark streaming
codecov-commenter edited a comment on pull request #3142: URL: https://github.com/apache/hudi/pull/3142#issuecomment-867078369 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3142](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f86f50e) into [master](https://codecov.io/gh/apache/hudi/commit/371526789d663dee85041eb31c27c52c81ef87ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3715267) will **decrease** coverage by `24.54%`. > The diff coverage is `0.00%`. [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/3142/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3142 +/- ## - Coverage 27.40% 2.86% -24.55% + Complexity 1287 85 -1202 Files 381 283 -98 Lines 15108 11710 -3398 Branches 1305 961 -344 - Hits 4141 335 -3806 - Misses10667 11349 +682 + Partials300 26 -274 ``` | Flag | Coverage Δ | | |---|---|---| | hudiclient | `0.00% <0.00%> (-21.06%)` | :arrow_down: | | hudisync | `5.37% <ø> (ø)` | | | hudiutilities | `9.11% <0.00%> (-49.46%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/async/AsyncClusteringService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/async/HoodieAsyncService.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/client/AbstractClusteringClient.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdENsdXN0ZXJpbmdDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/config/HoodieClusteringConfig.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVDbHVzdGVyaW5nQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3142/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Ap
[jira] [Commented] (HUDI-2138) Implement Parquest Data blocks for file inlining
[ https://issues.apache.org/jira/browse/HUDI-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378292#comment-17378292 ] ASF GitHub Bot commented on HUDI-2138: -- hudi-bot edited a comment on pull request #3228: URL: https://github.com/apache/hudi/pull/3228#issuecomment-874522164 ## CI report: * 5243cb2f0fc718cdb99434cffa7dc3da88c2b72b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=829) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Parquest Data blocks for file inlining > > > Key: HUDI-2138 > URL: https://issues.apache.org/jira/browse/HUDI-2138 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: performance, pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3228: [HUDI-2138] Add Parquet Log Block Formats
hudi-bot edited a comment on pull request #3228: URL: https://github.com/apache/hudi/pull/3228#issuecomment-874522164 ## CI report: * 5243cb2f0fc718cdb99434cffa7dc3da88c2b72b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=829) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2138) Implement Parquest Data blocks for file inlining
[ https://issues.apache.org/jira/browse/HUDI-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378291#comment-17378291 ] ASF GitHub Bot commented on HUDI-2138: -- hudi-bot edited a comment on pull request #3228: URL: https://github.com/apache/hudi/pull/3228#issuecomment-874522164 ## CI report: * d881b4e8cb27a3349fe07497246273d066437ca9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=769) * 5243cb2f0fc718cdb99434cffa7dc3da88c2b72b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=829) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Parquest Data blocks for file inlining > > > Key: HUDI-2138 > URL: https://issues.apache.org/jira/browse/HUDI-2138 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: performance, pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3228: [HUDI-2138] Add Parquet Log Block Formats
hudi-bot edited a comment on pull request #3228: URL: https://github.com/apache/hudi/pull/3228#issuecomment-874522164 ## CI report: * d881b4e8cb27a3349fe07497246273d066437ca9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=769) * 5243cb2f0fc718cdb99434cffa7dc3da88c2b72b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=829) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org