[jira] [Closed] (HUDI-1482) async clustering for spark streaming
[ https://issues.apache.org/jira/browse/HUDI-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-1482. - Resolution: Fixed > async clustering for spark streaming > > > Key: HUDI-1482 > URL: https://issues.apache.org/jira/browse/HUDI-1482 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: liwei >Assignee: Sagar Sumit >Priority: Blocker > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1138) Re-implement marker files via timeline server
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384665#comment-17384665 ] ASF GitHub Bot commented on HUDI-1138: -- codecov-commenter edited a comment on pull request #3233: URL: https://github.com/apache/hudi/pull/3233#issuecomment-883297547 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3233](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (230205e) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `20.20%`. > The diff coverage is `5.03%`. [](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3233 +/- ## = - Coverage 47.74% 27.53% -20.21% + Complexity 5591 1325 -4266 = Files 938 394 -544 Lines 4182315659-26164 Branches 4213 1384 -2829 = - Hits 19968 4312-15656 + Misses2007011023 -9047 + Partials 1785 324 -1461 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.07% <5.26%> (-13.49%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.78% <0.00%> (-0.10%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../client/embedded/EmbeddedTimelineServerHelper.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmVySGVscGVyLmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../hudi/client/embedded/EmbeddedTimelineService.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVdyaXRlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3233: [WIP][HUDI-1138] Add timeline-server-based marker file strategy for improving marker-related latency
codecov-commenter edited a comment on pull request #3233: URL: https://github.com/apache/hudi/pull/3233#issuecomment-883297547 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3233](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (230205e) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `20.20%`. > The diff coverage is `5.03%`. [](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3233 +/- ## = - Coverage 47.74% 27.53% -20.21% + Complexity 5591 1325 -4266 = Files 938 394 -544 Lines 4182315659-26164 Branches 4213 1384 -2829 = - Hits 19968 4312-15656 + Misses2007011023 -9047 + Partials 1785 324 -1461 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.07% <5.26%> (-13.49%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.78% <0.00%> (-0.10%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../client/embedded/EmbeddedTimelineServerHelper.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmVySGVscGVyLmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../hudi/client/embedded/EmbeddedTimelineService.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVdyaXRlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[jira] [Commented] (HUDI-1138) Re-implement marker files via timeline server
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384662#comment-17384662 ] ASF GitHub Bot commented on HUDI-1138: -- codecov-commenter edited a comment on pull request #3233: URL: https://github.com/apache/hudi/pull/3233#issuecomment-883297547 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3233](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (230205e) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `31.74%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3233 +/- ## = - Coverage 47.74% 15.99% -31.75% + Complexity 5591 505 -5086 = Files 938 288 -650 Lines 4182311953-29870 Branches 4213 989 -3224 = - Hits 19968 1912-18056 + Misses20070 9877-10193 + Partials 1785 164 -1621 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.78% <0.00%> (-0.10%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../client/embedded/EmbeddedTimelineServerHelper.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmVySGVscGVyLmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../hudi/client/embedded/EmbeddedTimelineService.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVdyaXRlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3233: [WIP][HUDI-1138] Add timeline-server-based marker file strategy for improving marker-related latency
codecov-commenter edited a comment on pull request #3233: URL: https://github.com/apache/hudi/pull/3233#issuecomment-883297547 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3233](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (230205e) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `31.74%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3233 +/- ## = - Coverage 47.74% 15.99% -31.75% + Complexity 5591 505 -5086 = Files 938 288 -650 Lines 4182311953-29870 Branches 4213 989 -3224 = - Hits 19968 1912-18056 + Misses20070 9877-10193 + Partials 1785 164 -1621 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.78% <0.00%> (-0.10%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../client/embedded/EmbeddedTimelineServerHelper.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmVySGVscGVyLmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../hudi/client/embedded/EmbeddedTimelineService.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVdyaXRlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[jira] [Resolved] (HUDI-2146) Concurrent writes loss data
[
https://issues.apache.org/jira/browse/HUDI-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar resolved HUDI-2146.
--
Resolution: Cannot Reproduce
> Concurrent writes loss data
>
>
> Key: HUDI-2146
> URL: https://issues.apache.org/jira/browse/HUDI-2146
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Wenning Ding
>Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: image-2021-07-08-00-49-30-730.png
>
>
> Reproduction steps:
> Create a Hudi table:
> {code:java}
> import org.apache.hudi.DataSourceWriteOptions
> import org.apache.hudi.config.HoodieWriteConfig
> import org.apache.spark.sql.SaveMode
> import org.apache.hudi.AvroConversionUtils
> val df = Seq(
> (100, "event_name_16", "2015-01-01T13:51:39.340396Z", "type1"),
> (101, "event_name_546", "2015-01-01T12:14:58.597216Z", "type2"),
> (104, "event_name_123", "2015-01-01T12:15:00.512679Z", "type1"),
> (105, "event_name_678", "2015-01-01T13:51:42.248818Z", "type2")
> ).toDF("event_id", "event_name", "event_ts", "event_type")
> var tableName = "hudi_test"
> var tablePath = "s3://.../" + tableName
> // write hudi dataset
> df.write.format("org.apache.hudi")
> .option(HoodieWriteConfig.TABLE_NAME, tableName)
> .option(DataSourceWriteOptions.OPERATION_OPT_KEY,
> DataSourceWriteOptions.INSERT_OPERATION_OPT_VAL)
> .option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY,
> DataSourceWriteOptions.COW_STORAGE_TYPE_OPT_VAL)
> .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "event_id")
> .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "event_type")
> .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "event_ts")
> .option(DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY, "true")
> .option(DataSourceWriteOptions.HIVE_TABLE_OPT_KEY, tableName)
> .option(DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY, "event_type")
> .option(DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,
> "org.apache.hudi.hive.MultiPartKeysValueExtractor")
> .mode(SaveMode.Overwrite)
> .save(tablePath)
> {code}
> Perform two insert operations almost in the same time, each insertion
> contains different data:
> Insert 1:
> {code:java}
> val df3 = Seq(
> (400, "event_name_11", "2125-02-01T13:51:39.340396Z", "type1"),
> (401, "event_name_22", "2125-02-01T12:14:58.597216Z", "type2"),
> (404, "event_name_333433", "2126-01-01T12:15:00.512679Z", "type1"),
> (405, "event_name_666378", "2125-07-01T13:51:42.248818Z", "type2")
> ).toDF("event_id", "event_name", "event_ts", "event_type")
> // update hudi dataset
> df3.write.format("org.apache.hudi")
>.option(HoodieWriteConfig.TABLE_NAME, tableName)
>.option(DataSourceWriteOptions.OPERATION_OPT_KEY,
> DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL)
>.option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY,
> DataSourceWriteOptions.COW_STORAGE_TYPE_OPT_VAL)
>.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "event_id")
>.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "event_type")
>.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "event_ts")
>.option(DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY, "true")
>.option(DataSourceWriteOptions.HIVE_TABLE_OPT_KEY, tableName)
>.option(DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY, "event_type")
>.option(DataSourceWriteOptions.HIVE_ASSUME_DATE_PARTITION_OPT_KEY, "false")
>.option(DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,
> "org.apache.hudi.hive.MultiPartKeysValueExtractor")
>.option("hoodie.write.concurrency.mode", "optimistic_concurrency_control")
>.option("hoodie.cleaner.policy.failed.writes", "LAZY")
>.option("hoodie.write.lock.provider",
> "org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider")
>.option("hoodie.write.lock.zookeeper.url", "ip-***.ec2.internal")
>.option("hoodie.write.lock.zookeeper.port", "2181")
>.option("hoodie.write.lock.zookeeper.lock_key", tableName)
>.option("hoodie.write.lock.zookeeper.base_path", "/occ_lock")
>.mode(SaveMode.Append)
>.save(tablePath)
> {code}
> Insert 2:
> {code:java}
> val df3 = Seq(
> (300, "event_name_1", "2035-02-01T13:51:39.340396Z", "type1"),
> (301, "event_name_2", "2035-02-01T12:14:58.597216Z", "type2"),
> (304, "event_name_3", "2036-01-01T12:15:00.512679Z", "type1"),
> (305, "event_name_66678", "2035-07-01T13:51:42.248818Z", "type2")
> ).toDF("event_id", "event_name", "event_ts", "event_type")
> // update hudi dataset
> df3.write.format("org.apache.hudi")
>.option(HoodieWriteConfig.TABLE_NAME, tableName)
>.option(DataSourceWriteOptions.OPERATION_OPT_KEY,
> DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL)
>.option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY,
>
[hudi] branch master updated (d58a834 -> 4f1350f)
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from d58a834 [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node (#3074) add 4f1350f [MINOR] Disable codecov (#3314) No new revisions were added by this update. Summary of changes: .codecov.yml.broken => .codecov.yml | 16 1 file changed, 8 insertions(+), 8 deletions(-) rename .codecov.yml.broken => .codecov.yml (89%)
[GitHub] [hudi] vinothchandar merged pull request #3314: [MINOR] Disable codecov
vinothchandar merged pull request #3314: URL: https://github.com/apache/hudi/pull/3314 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1138) Re-implement marker files via timeline server
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384652#comment-17384652 ] ASF GitHub Bot commented on HUDI-1138: -- codecov-commenter edited a comment on pull request #3233: URL: https://github.com/apache/hudi/pull/3233#issuecomment-883297547 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3233](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (230205e) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `44.94%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3233 +/- ## - Coverage 47.74% 2.80% -44.95% + Complexity 5591 85 -5506 Files 938 288 -650 Lines 41823 11953-29870 Branches 4213 989 -3224 - Hits 19968 335-19633 + Misses20070 11592 -8478 + Partials 1785 26 -1759 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.95% <0.00%> (-50.92%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../client/embedded/EmbeddedTimelineServerHelper.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmVySGVscGVyLmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../hudi/client/embedded/EmbeddedTimelineService.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVdyaXRlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3233: [WIP][HUDI-1138] Add timeline-server-based marker file strategy for improving marker-related latency
codecov-commenter edited a comment on pull request #3233: URL: https://github.com/apache/hudi/pull/3233#issuecomment-883297547 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3233](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (230205e) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `44.94%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3233 +/- ## - Coverage 47.74% 2.80% -44.95% + Complexity 5591 85 -5506 Files 938 288 -650 Lines 41823 11953-29870 Branches 4213 989 -3224 - Hits 19968 335-19633 + Misses20070 11592 -8478 + Partials 1785 26 -1759 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.95% <0.00%> (-50.92%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3233?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../client/embedded/EmbeddedTimelineServerHelper.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmVySGVscGVyLmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../hudi/client/embedded/EmbeddedTimelineService.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9lbWJlZGRlZC9FbWJlZGRlZFRpbWVsaW5lU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/hudi/pull/3233/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVdyaXRlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[jira] [Commented] (HUDI-2139) MergeInto MOR Table May Result InCorrect Result
[
https://issues.apache.org/jira/browse/HUDI-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384647#comment-17384647
]
ASF GitHub Bot commented on HUDI-2139:
--
hudi-bot edited a comment on pull request #3230:
URL: https://github.com/apache/hudi/pull/3230#issuecomment-874605269
## CI report:
* 2568538186a80a3cea8470e01cafb654a825c75a UNKNOWN
* 2b6219dd1b9a223dcff9e37548a14e5e1dad9813 Azure:
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1065)
Bot commands
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MergeInto MOR Table May Result InCorrect Result
> ---
>
> Key: HUDI-2139
> URL: https://issues.apache.org/jira/browse/HUDI-2139
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Currently we process all the update-action and inert-action in the
> ExpressionPayload#
> getInsertValue without know whether the record is matched or not matched for
> MOR table. This may result in incorrect merge result. e.g.
> {code:java}
> Merge into h0
> using (select 2 as id, 'a1' as name, 10 as price from s) s0
> on h0.id = s0.id
> when matched then s0.id = 1 the update set id = s0.id, name = s0.name, price
> = 10
> when not matched then s0.id = 2 the insert (id,name,price) values(id,name,
> 20){code}
> If the id = 2 can matched the target table h0, but it cannot match the
> udpate-condition ( s0.id = 1), It should not update the table. However,
> currently we cannot know the matched state of the input record, it will goes
> to the not-matched actions and update the price to 20 finally. This is
> incorrect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3230: [HUDI-2139] MergeInto MOR Table May Result InCorrect Result
hudi-bot edited a comment on pull request #3230: URL: https://github.com/apache/hudi/pull/3230#issuecomment-874605269 ## CI report: * 2568538186a80a3cea8470e01cafb654a825c75a UNKNOWN * 2b6219dd1b9a223dcff9e37548a14e5e1dad9813 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1065) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HUDI-1563) Documentation on small file handling
[ https://issues.apache.org/jira/browse/HUDI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1563. --- Fix Version/s: 0.8.0 Resolution: Fixed > Documentation on small file handling > > > Key: HUDI-1563 > URL: https://issues.apache.org/jira/browse/HUDI-1563 > Project: Apache Hudi > Issue Type: Improvement > Components: Docs >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available, user-support-issues > Fix For: 0.8.0 > > > Questions from slack: > how does Hudi handle small files. What all config knobs one has to play > around w/. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1378) Add ENABLE_ROW_WRITER_OPT_KEY to Configurations (docs)
[ https://issues.apache.org/jira/browse/HUDI-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1378: -- Status: In Progress (was: Open) > Add ENABLE_ROW_WRITER_OPT_KEY to Configurations (docs) > -- > > Key: HUDI-1378 > URL: https://issues.apache.org/jira/browse/HUDI-1378 > Project: Apache Hudi > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Fix For: 0.9.0 > > > hoodie.datasource.write.row.writer.enable > > /** > * When set to true, will perform write operations directly using the spark > native `Row` representation. > * By default, false (will be enabled as default in a future release) > */ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1378) Add ENABLE_ROW_WRITER_OPT_KEY to Configurations (docs)
[ https://issues.apache.org/jira/browse/HUDI-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1378. --- Resolution: Fixed > Add ENABLE_ROW_WRITER_OPT_KEY to Configurations (docs) > -- > > Key: HUDI-1378 > URL: https://issues.apache.org/jira/browse/HUDI-1378 > Project: Apache Hudi > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Fix For: 0.9.0 > > > hoodie.datasource.write.row.writer.enable > > /** > * When set to true, will perform write operations directly using the spark > native `Row` representation. > * By default, false (will be enabled as default in a future release) > */ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1563) Documentation on small file handling
[ https://issues.apache.org/jira/browse/HUDI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1563: -- Status: In Progress (was: Open) > Documentation on small file handling > > > Key: HUDI-1563 > URL: https://issues.apache.org/jira/browse/HUDI-1563 > Project: Apache Hudi > Issue Type: Improvement > Components: Docs >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available, user-support-issues > > Questions from slack: > how does Hudi handle small files. What all config knobs one has to play > around w/. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1378) Add ENABLE_ROW_WRITER_OPT_KEY to Configurations (docs)
[ https://issues.apache.org/jira/browse/HUDI-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1378: -- Status: Open (was: New) > Add ENABLE_ROW_WRITER_OPT_KEY to Configurations (docs) > -- > > Key: HUDI-1378 > URL: https://issues.apache.org/jira/browse/HUDI-1378 > Project: Apache Hudi > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Fix For: 0.9.0 > > > hoodie.datasource.write.row.writer.enable > > /** > * When set to true, will perform write operations directly using the spark > native `Row` representation. > * By default, false (will be enabled as default in a future release) > */ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-1838) Add documentation for using cleaning / archiving configs
[ https://issues.apache.org/jira/browse/HUDI-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-1838. - Resolution: Duplicate > Add documentation for using cleaning / archiving configs > > > Key: HUDI-1838 > URL: https://issues.apache.org/jira/browse/HUDI-1838 > Project: Apache Hudi > Issue Type: Bug > Components: configs >Reporter: Nishith Agarwal >Assignee: sivabalan narayanan >Priority: Major > Labels: sev:normal > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-1106) Bulk insert dataset - Drop duplicates
[ https://issues.apache.org/jira/browse/HUDI-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-1106. - Resolution: Duplicate > Bulk insert dataset - Drop duplicates > - > > Key: HUDI-1106 > URL: https://issues.apache.org/jira/browse/HUDI-1106 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Affects Versions: 0.9.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1941) Fix callers of HoodieRecordPayload.preCombine() to use new api with props arg
[ https://issues.apache.org/jira/browse/HUDI-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1941: -- Labels: sev:critical (was: sev:high) > Fix callers of HoodieRecordPayload.preCombine() to use new api with props arg > - > > Key: HUDI-1941 > URL: https://issues.apache.org/jira/browse/HUDI-1941 > Project: Apache Hudi > Issue Type: Task > Components: Writer Core >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: sev:critical > > We deprecated old api for preCombine and introduced new one. But haven't > fixed the callers to use the new api. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1106) Bulk insert dataset - Drop duplicates
[ https://issues.apache.org/jira/browse/HUDI-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1106: -- Status: Open (was: New) > Bulk insert dataset - Drop duplicates > - > > Key: HUDI-1106 > URL: https://issues.apache.org/jira/browse/HUDI-1106 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Affects Versions: 0.9.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1991) Fix drop dups flow for bulk insert in row writer path
[ https://issues.apache.org/jira/browse/HUDI-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1991: -- Status: In Progress (was: Open) > Fix drop dups flow for bulk insert in row writer path > - > > Key: HUDI-1991 > URL: https://issues.apache.org/jira/browse/HUDI-1991 > Project: Apache Hudi > Issue Type: Task >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Fix drop dups flow for bulk insert in row writer path. > If enabled, throw exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1991) Fix drop dups flow for bulk insert in row writer path
[ https://issues.apache.org/jira/browse/HUDI-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1991. --- Fix Version/s: 0.9.0 Resolution: Fixed > Fix drop dups flow for bulk insert in row writer path > - > > Key: HUDI-1991 > URL: https://issues.apache.org/jira/browse/HUDI-1991 > Project: Apache Hudi > Issue Type: Task >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Fix drop dups flow for bulk insert in row writer path. > If enabled, throw exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2095) Scope out Append_only mode for hudi
[ https://issues.apache.org/jira/browse/HUDI-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2095: -- Status: In Progress (was: Open) > Scope out Append_only mode for hudi > --- > > Key: HUDI-2095 > URL: https://issues.apache.org/jira/browse/HUDI-2095 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > > If use-case is just append only, we can reduce the meta columns overhead in > hudi and directly store the incoming df as is. So, this task explores the > path towards supporting this and scopes out what works and what may not work > in hudi to support this. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2006) Add more yamls to test suite
[ https://issues.apache.org/jira/browse/HUDI-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2006: -- Status: In Progress (was: Open) > Add more yamls to test suite > - > > Key: HUDI-2006 > URL: https://issues.apache.org/jira/browse/HUDI-2006 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Add more yaml files to test suite job suite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-2095) Scope out Append_only mode for hudi
[ https://issues.apache.org/jira/browse/HUDI-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2095. --- Fix Version/s: 0.9.0 Resolution: Fixed > Scope out Append_only mode for hudi > --- > > Key: HUDI-2095 > URL: https://issues.apache.org/jira/browse/HUDI-2095 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Fix For: 0.9.0 > > > If use-case is just append only, we can reduce the meta columns overhead in > hudi and directly store the incoming df as is. So, this task explores the > path towards supporting this and scopes out what works and what may not work > in hudi to support this. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1104) Bulk insert Dataset - UserDefinedPartitioner
[ https://issues.apache.org/jira/browse/HUDI-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1104: -- Status: In Progress (was: Open) > Bulk insert Dataset - UserDefinedPartitioner > > > Key: HUDI-1104 > URL: https://issues.apache.org/jira/browse/HUDI-1104 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1104) Bulk insert Dataset - UserDefinedPartitioner
[ https://issues.apache.org/jira/browse/HUDI-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1104. --- Resolution: Fixed > Bulk insert Dataset - UserDefinedPartitioner > > > Key: HUDI-1104 > URL: https://issues.apache.org/jira/browse/HUDI-1104 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-2006) Add more yamls to test suite
[ https://issues.apache.org/jira/browse/HUDI-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2006. --- Fix Version/s: 0.9.0 Resolution: Fixed > Add more yamls to test suite > - > > Key: HUDI-2006 > URL: https://issues.apache.org/jira/browse/HUDI-2006 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Add more yaml files to test suite job suite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1104) Bulk insert Dataset - UserDefinedPartitioner
[ https://issues.apache.org/jira/browse/HUDI-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1104: -- Status: Open (was: New) > Bulk insert Dataset - UserDefinedPartitioner > > > Key: HUDI-1104 > URL: https://issues.apache.org/jira/browse/HUDI-1104 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1105) Bulk insert dataset - Dedup
[ https://issues.apache.org/jira/browse/HUDI-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1105: -- Status: Open (was: New) > Bulk insert dataset - Dedup > --- > > Key: HUDI-1105 > URL: https://issues.apache.org/jira/browse/HUDI-1105 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Affects Versions: 0.9.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1105) Bulk insert dataset - Dedup
[ https://issues.apache.org/jira/browse/HUDI-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1105: -- Status: In Progress (was: Open) > Bulk insert dataset - Dedup > --- > > Key: HUDI-1105 > URL: https://issues.apache.org/jira/browse/HUDI-1105 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Affects Versions: 0.9.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1105) Bulk insert dataset - Dedup
[ https://issues.apache.org/jira/browse/HUDI-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1105. --- Resolution: Fixed > Bulk insert dataset - Dedup > --- > > Key: HUDI-1105 > URL: https://issues.apache.org/jira/browse/HUDI-1105 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Affects Versions: 0.9.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-2009) Fix extra commit metadata in row writer path
[ https://issues.apache.org/jira/browse/HUDI-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2009. --- Fix Version/s: 0.9.0 Resolution: Fixed > Fix extra commit metadata in row writer path > > > Key: HUDI-2009 > URL: https://issues.apache.org/jira/browse/HUDI-2009 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > In regular path (write client), users can pass in extra commit metadata with > help of a commit key prefix config. In row writer path, this was not > addressed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2009) Fix extra commit metadata in row writer path
[ https://issues.apache.org/jira/browse/HUDI-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2009: -- Status: In Progress (was: Open) > Fix extra commit metadata in row writer path > > > Key: HUDI-2009 > URL: https://issues.apache.org/jira/browse/HUDI-2009 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > In regular path (write client), users can pass in extra commit metadata with > help of a commit key prefix config. In row writer path, this was not > addressed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2007) Adding spark bulk insert node
[ https://issues.apache.org/jira/browse/HUDI-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2007: -- Status: In Progress (was: Open) > Adding spark bulk insert node > - > > Key: HUDI-2007 > URL: https://issues.apache.org/jira/browse/HUDI-2007 > Project: Apache Hudi > Issue Type: Improvement > Components: Testing >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Adding spark bulk insert node and add config to enable row writing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-2007) Adding spark bulk insert node
[ https://issues.apache.org/jira/browse/HUDI-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2007. --- Fix Version/s: 0.9.0 Resolution: Fixed > Adding spark bulk insert node > - > > Key: HUDI-2007 > URL: https://issues.apache.org/jira/browse/HUDI-2007 > Project: Apache Hudi > Issue Type: Improvement > Components: Testing >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Adding spark bulk insert node and add config to enable row writing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2177) Virtual keys support for Compaction
[ https://issues.apache.org/jira/browse/HUDI-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384645#comment-17384645 ] ASF GitHub Bot commented on HUDI-2177: -- hudi-bot edited a comment on pull request #3315: URL: https://github.com/apache/hudi/pull/3315#issuecomment-883851530 ## CI report: * b4b137968fc3ceed09408fdef934bba73764c5e5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1064) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virtual keys support for Compaction > --- > > Key: HUDI-2177 > URL: https://issues.apache.org/jira/browse/HUDI-2177 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virtual keys support for Compaction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3315: [HUDI-2177][HUDI-2200] Adding virtual keys support for MOR
hudi-bot edited a comment on pull request #3315: URL: https://github.com/apache/hudi/pull/3315#issuecomment-883851530 ## CI report: * b4b137968fc3ceed09408fdef934bba73764c5e5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1064) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-2202) Add Trino to Docker Demo
Sagar Sumit created HUDI-2202: - Summary: Add Trino to Docker Demo Key: HUDI-2202 URL: https://issues.apache.org/jira/browse/HUDI-2202 Project: Apache Hudi Issue Type: Sub-task Reporter: Sagar Sumit Assignee: Sagar Sumit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1978) [UMBRELLA] Support for Trino
[ https://issues.apache.org/jira/browse/HUDI-1978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-1978: - Assignee: Sagar Sumit (was: Vinoth Chandar) > [UMBRELLA] Support for Trino > > > Key: HUDI-1978 > URL: https://issues.apache.org/jira/browse/HUDI-1978 > Project: Apache Hudi > Issue Type: New Feature > Components: trino >Reporter: Vinoth Chandar >Assignee: Sagar Sumit >Priority: Major > Labels: hudi-umbrellas > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1856) Upstream changes made in PrestoDB to eliminate file listing to Trino
[ https://issues.apache.org/jira/browse/HUDI-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-1856: - Assignee: Sagar Sumit > Upstream changes made in PrestoDB to eliminate file listing to Trino > > > Key: HUDI-1856 > URL: https://issues.apache.org/jira/browse/HUDI-1856 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Nishith Agarwal >Assignee: Sagar Sumit >Priority: Blocker > Labels: sev:high, sev:triage > Fix For: 0.9.0 > > > inputFormat.getSplits() code was optimized for PrestoDB code base. This > change is not implemented / upstreamed in Trino. > > Additionally, there are other changes that need to be upstreamed in Trino. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2158) Upstream support for MOR tables.
[ https://issues.apache.org/jira/browse/HUDI-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-2158: - Assignee: Sagar Sumit > Upstream support for MOR tables. > > > Key: HUDI-2158 > URL: https://issues.apache.org/jira/browse/HUDI-2158 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Vinoth Chandar >Assignee: Sagar Sumit >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1138) Re-implement marker files via timeline server
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384640#comment-17384640 ] ASF GitHub Bot commented on HUDI-1138: -- hudi-bot edited a comment on pull request #3233: URL: https://github.com/apache/hudi/pull/3233#issuecomment-875280958 ## CI report: * 2d22335c215ed620ce20018b1c83be189b7c70c6 UNKNOWN * f1095198d43636de20b525d80341c32f84591d48 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1042) * 230205edfab190cfaf687d0323ae8d704f425e1d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Re-implement marker files via timeline server > - > > Key: HUDI-1138 > URL: https://issues.apache.org/jira/browse/HUDI-1138 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Affects Versions: 0.9.0 >Reporter: Vinoth Chandar >Assignee: Ethan Guo >Priority: Blocker > Labels: pull-request-available > Fix For: 0.9.0 > > > Even as you can argue that RFC-15/consolidated metadata, removes the need for > deleting partial files written due to spark task failures/stage retries. It > will still leave extra files inside the table (and users will pay for it > every month) and we need the marker mechanism to be able to delete these > partial files. > Here we explore if we can improve the current marker file mechanism, that > creates one marker file per data file written, by > Delegating the createMarker() call to the driver/timeline server, and have it > create marker metadata into a single file handle, that is flushed for > durability guarantees > > P.S: I was tempted to think Spark listener mechanism can help us deal with > failed tasks, but it has no guarantees. the writer job could die without > deleting a partial file. i.e it can improve things, but cant provide > guarantees -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3233: [WIP][HUDI-1138] Add timeline-server-based marker file strategy for improving marker-related latency
hudi-bot edited a comment on pull request #3233: URL: https://github.com/apache/hudi/pull/3233#issuecomment-875280958 ## CI report: * 2d22335c215ed620ce20018b1c83be189b7c70c6 UNKNOWN * f1095198d43636de20b525d80341c32f84591d48 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1042) * 230205edfab190cfaf687d0323ae8d704f425e1d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2007) Adding spark bulk insert node
[ https://issues.apache.org/jira/browse/HUDI-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384638#comment-17384638 ] ASF GitHub Bot commented on HUDI-2007: -- nsivabalan merged pull request #3074: URL: https://github.com/apache/hudi/pull/3074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Adding spark bulk insert node > - > > Key: HUDI-2007 > URL: https://issues.apache.org/jira/browse/HUDI-2007 > Project: Apache Hudi > Issue Type: Improvement > Components: Testing >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Adding spark bulk insert node and add config to enable row writing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[hudi] branch master updated (858e84b -> d58a834)
This is an automated email from the ASF dual-hosted git repository.
sivabalan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git.
from 858e84b [HUDI-2198] Clean and reset the bootstrap events for
coordinator when task failover (#3304)
add d58a834 [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding
spark bulk_insert node (#3074)
No new revisions were added by this update.
Summary of changes:
hudi-integ-test/pom.xml| 72 ++
.../integ/testsuite/configuration/DeltaConfig.java | 5 ++
...kUpsertNode.scala => SparkBulkInsertNode.scala} | 19 +++---
3 files changed, 87 insertions(+), 9 deletions(-)
copy
hudi-integ-test/src/main/scala/org/apache/hudi/integ/testsuite/dag/nodes/{SparkUpsertNode.scala
=> SparkBulkInsertNode.scala} (74%)
[GitHub] [hudi] nsivabalan merged pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
nsivabalan merged pull request #3074: URL: https://github.com/apache/hudi/pull/3074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1518) Remove replaced files logic from archival
[ https://issues.apache.org/jira/browse/HUDI-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384634#comment-17384634 ] ASF GitHub Bot commented on HUDI-1518: -- codecov-commenter edited a comment on pull request #3310: URL: https://github.com/apache/hudi/pull/3310#issuecomment-883331907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove replaced files logic from archival > - > > Key: HUDI-1518 > URL: https://issues.apache.org/jira/browse/HUDI-1518 > Project: Apache Hudi > Issue Type: Task >Affects Versions: 0.9.0 >Reporter: satish >Priority: Minor > Labels: pull-request-available > Fix For: 0.9.0 > > > See > https://github.com/apache/hudi/blob/79ec7b4894b997183a6e10fdc19d34f5ab4ea437/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java#L324 > Archival tries to delete replaced files in 'deleteReplacedFileGroups' method. > We changed cleaner to delete replaced files. So this is no longer needed. We > can instead validate that replaced files no longer exist on disk during > archival. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3307](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (5ad4271) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `30.19%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3307 +/- ## = - Coverage 47.74% 17.54% -30.20% + Complexity 5591 905 -4686 = Files 938 390 -548 Lines 4182315587-26236 Branches 4213 1381 -2832 = - Hits 19968 2735-17233 + Misses2007012666 -7404 + Partials 1785 186 -1599 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.19% <ø> (-13.36%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.91%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3310: [HUDI-1518] Remove the logic that delete replaced file when archive
codecov-commenter edited a comment on pull request #3310: URL: https://github.com/apache/hudi/pull/3310#issuecomment-883331907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2007) Adding spark bulk insert node
[ https://issues.apache.org/jira/browse/HUDI-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384633#comment-17384633 ] ASF GitHub Bot commented on HUDI-2007: -- codecov-commenter edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-860337193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Adding spark bulk insert node > - > > Key: HUDI-2007 > URL: https://issues.apache.org/jira/browse/HUDI-2007 > Project: Apache Hudi > Issue Type: Improvement > Components: Testing >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Adding spark bulk insert node and add config to enable row writing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
codecov-commenter edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-860337193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
hudi-bot edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883153186 ## CI report: * 5ad4271d6c2cf000bc195d3945cbe2dd0f6cbb4a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1062) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2007) Adding spark bulk insert node
[ https://issues.apache.org/jira/browse/HUDI-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384629#comment-17384629 ] ASF GitHub Bot commented on HUDI-2007: -- codecov-commenter edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-860337193 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3074](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9c622e5) into [master](https://codecov.io/gh/apache/hudi/commit/634163a990569aa4463b58830396f455dd15340c?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (634163a) will **decrease** coverage by `20.08%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3074 +/- ## = - Coverage 47.74% 27.66% -20.09% + Complexity 5592 1325 -4267 = Files 938 390 -548 Lines 4182515587-26238 Branches 4213 1381 -2832 = - Hits 19971 4312-15659 + Misses2007110951 -9120 + Partials 1783 324 -1459 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.19% <ø> (-13.36%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
codecov-commenter edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-860337193 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3074](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9c622e5) into [master](https://codecov.io/gh/apache/hudi/commit/634163a990569aa4463b58830396f455dd15340c?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (634163a) will **decrease** coverage by `20.08%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3074 +/- ## = - Coverage 47.74% 27.66% -20.09% + Complexity 5592 1325 -4267 = Files 938 390 -548 Lines 4182515587-26238 Branches 4213 1381 -2832 = - Hits 19971 4312-15659 + Misses2007110951 -9120 + Partials 1783 324 -1459 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.19% <ø> (-13.36%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[jira] [Commented] (HUDI-1518) Remove replaced files logic from archival
[ https://issues.apache.org/jira/browse/HUDI-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384628#comment-17384628 ] ASF GitHub Bot commented on HUDI-1518: -- codecov-commenter edited a comment on pull request #3310: URL: https://github.com/apache/hudi/pull/3310#issuecomment-883331907 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3310](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f0b369b) into [master](https://codecov.io/gh/apache/hudi/commit/572a2144124b90b2dbef8183ee19a95b21de2716?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (572a214) will **decrease** coverage by `20.14%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3310 +/- ## = - Coverage 47.83% 27.68% -20.15% + Complexity 5565 1325 -4240 = Files 936 390 -546 Lines 4166315574-26089 Branches 4197 1380 -2817 = - Hits 19929 4312-15617 + Misses1996010938 -9022 + Partials 1774 324 -1450 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.22% <ø> (-13.31%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (+0.10%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/HoodieTimelineArchiveLog.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL0hvb2RpZVRpbWVsaW5lQXJjaGl2ZUxvZy5qYXZh) | `0.00% <ø> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3310: [HUDI-1518] Remove the logic that delete replaced file when archive
codecov-commenter edited a comment on pull request #3310: URL: https://github.com/apache/hudi/pull/3310#issuecomment-883331907 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3310](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f0b369b) into [master](https://codecov.io/gh/apache/hudi/commit/572a2144124b90b2dbef8183ee19a95b21de2716?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (572a214) will **decrease** coverage by `20.14%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3310 +/- ## = - Coverage 47.83% 27.68% -20.15% + Complexity 5565 1325 -4240 = Files 936 390 -546 Lines 4166315574-26089 Branches 4197 1380 -2817 = - Hits 19929 4312-15617 + Misses1996010938 -9022 + Partials 1774 324 -1450 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.22% <ø> (-13.31%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (+0.10%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/HoodieTimelineArchiveLog.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL0hvb2RpZVRpbWVsaW5lQXJjaGl2ZUxvZy5qYXZh) | `0.00% <ø> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[jira] [Updated] (HUDI-2138) Implement Parquet Data blocks for file inlining
[ https://issues.apache.org/jira/browse/HUDI-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2138: - Summary: Implement Parquet Data blocks for file inlining (was: Implement Parquest Data blocks for file inlining) > Implement Parquet Data blocks for file inlining > --- > > Key: HUDI-2138 > URL: https://issues.apache.org/jira/browse/HUDI-2138 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Rajesh Mahindra >Assignee: Rajesh Mahindra >Priority: Major > Labels: performance, pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2007) Adding spark bulk insert node
[ https://issues.apache.org/jira/browse/HUDI-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384625#comment-17384625 ] ASF GitHub Bot commented on HUDI-2007: -- hudi-bot edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-861653947 ## CI report: * 9c622e581cf38a62828f734514cf6204d902ac76 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1061) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1060) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Adding spark bulk insert node > - > > Key: HUDI-2007 > URL: https://issues.apache.org/jira/browse/HUDI-2007 > Project: Apache Hudi > Issue Type: Improvement > Components: Testing >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Adding spark bulk insert node and add config to enable row writing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
hudi-bot edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-861653947 ## CI report: * 9c622e581cf38a62828f734514cf6204d902ac76 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1061) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1060) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2007) Adding spark bulk insert node
[ https://issues.apache.org/jira/browse/HUDI-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384624#comment-17384624 ] ASF GitHub Bot commented on HUDI-2007: -- codecov-commenter edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-860337193 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3074](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9c622e5) into [master](https://codecov.io/gh/apache/hudi/commit/634163a990569aa4463b58830396f455dd15340c?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (634163a) will **decrease** coverage by `31.65%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3074 +/- ## = - Coverage 47.74% 16.09% -31.66% + Complexity 5592 505 -5087 = Files 938 284 -654 Lines 4182511881-29944 Branches 4213 986 -3227 = - Hits 19971 1912-18059 + Misses20071 9805-10266 + Partials 1783 164 -1619 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
codecov-commenter edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-860337193 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3074](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9c622e5) into [master](https://codecov.io/gh/apache/hudi/commit/634163a990569aa4463b58830396f455dd15340c?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (634163a) will **decrease** coverage by `31.65%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3074 +/- ## = - Coverage 47.74% 16.09% -31.66% + Complexity 5592 505 -5087 = Files 938 284 -654 Lines 4182511881-29944 Branches 4213 986 -3227 = - Hits 19971 1912-18059 + Misses20071 9805-10266 + Partials 1783 164 -1619 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[jira] [Commented] (HUDI-1518) Remove replaced files logic from archival
[ https://issues.apache.org/jira/browse/HUDI-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384622#comment-17384622 ] ASF GitHub Bot commented on HUDI-1518: -- codecov-commenter edited a comment on pull request #3310: URL: https://github.com/apache/hudi/pull/3310#issuecomment-883331907 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3310](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f0b369b) into [master](https://codecov.io/gh/apache/hudi/commit/572a2144124b90b2dbef8183ee19a95b21de2716?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (572a214) will **decrease** coverage by `31.72%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3310 +/- ## = - Coverage 47.83% 16.11% -31.73% + Complexity 5565 505 -5060 = Files 936 284 -652 Lines 4166311868-29795 Branches 4197 985 -3212 = - Hits 19929 1912-18017 + Misses19960 9792-10168 + Partials 1774 164 -1610 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (+0.10%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/HoodieTimelineArchiveLog.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL0hvb2RpZVRpbWVsaW5lQXJjaGl2ZUxvZy5qYXZh) | `0.00% <ø> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3310: [HUDI-1518] Remove the logic that delete replaced file when archive
codecov-commenter edited a comment on pull request #3310: URL: https://github.com/apache/hudi/pull/3310#issuecomment-883331907 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3310](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f0b369b) into [master](https://codecov.io/gh/apache/hudi/commit/572a2144124b90b2dbef8183ee19a95b21de2716?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (572a214) will **decrease** coverage by `31.72%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3310 +/- ## = - Coverage 47.83% 16.11% -31.73% + Complexity 5565 505 -5060 = Files 936 284 -652 Lines 4166311868-29795 Branches 4197 985 -3212 = - Hits 19929 1912-18017 + Misses19960 9792-10168 + Partials 1774 164 -1610 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (+0.10%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/HoodieTimelineArchiveLog.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL0hvb2RpZVRpbWVsaW5lQXJjaGl2ZUxvZy5qYXZh) | `0.00% <ø> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[jira] [Commented] (HUDI-2177) Virtual keys support for Compaction
[ https://issues.apache.org/jira/browse/HUDI-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384621#comment-17384621 ] ASF GitHub Bot commented on HUDI-2177: -- codecov-commenter commented on pull request #3315: URL: https://github.com/apache/hudi/pull/3315#issuecomment-883856272 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3315?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3315](https://codecov.io/gh/apache/hudi/pull/3315?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b4b1379) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `44.91%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3315?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3315 +/- ## - Coverage 47.74% 2.82% -44.92% + Complexity 5591 85 -5506 Files 938 280 -658 Lines 41823 11862-29961 Branches 4213 989 -3224 - Hits 19968 335-19633 + Misses20070 11501 -8569 + Partials 1785 26 -1759 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.91%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3315?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3315/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3315/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/io/HoodieKeyLocationFetchHandle.java](https://codecov.io/gh/apache/hudi/pull/3315/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUtleUxvY2F0aW9uRmV0Y2hIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3315/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/io/HoodieSortedMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3315/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVNvcnRlZE1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[GitHub] [hudi] codecov-commenter commented on pull request #3315: [HUDI-2177][HUDI-2200] Adding virtual keys support for MOR
codecov-commenter commented on pull request #3315: URL: https://github.com/apache/hudi/pull/3315#issuecomment-883856272 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3315?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3315](https://codecov.io/gh/apache/hudi/pull/3315?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b4b1379) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `44.91%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3315?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3315 +/- ## - Coverage 47.74% 2.82% -44.92% + Complexity 5591 85 -5506 Files 938 280 -658 Lines 41823 11862-29961 Branches 4213 989 -3224 - Hits 19968 335-19633 + Misses20070 11501 -8569 + Partials 1785 26 -1759 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.91%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3315?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3315/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3315/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/io/HoodieKeyLocationFetchHandle.java](https://codecov.io/gh/apache/hudi/pull/3315/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUtleUxvY2F0aW9uRmV0Y2hIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3315/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/io/HoodieSortedMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3315/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVNvcnRlZE1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[jira] [Updated] (HUDI-2201) Audi all flink classes and route all FileSystem list and getStatus calls to metadata table
[
https://issues.apache.org/jira/browse/HUDI-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-2201:
--
Description:
{code:java}
grep -irl "fs.listStatus" hudi-*/* | grep -v Test
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
{code}
{code:java}
grep -irl "fs.getFileStatus" hudi-*/* | grep -v Test
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkMergeHandle.java
hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/profile/WriteProfiles.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
{code}
Some of them are valid ones which can't be routed. But just dumping all matches
here.
was:
{code:java}
grep -irl "fs.listStatus" hudi-*/* | grep -v Test
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieLogFileCommand.java
hudi-cli/src/main/scala/org/apache/hudi/cli/DedupeSparkJob.scala
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/MarkerFiles.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/rollback/JavaListingBasedRollbackHelper.java
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FailSafeConsistencyGuard.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateAsyncOperations.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateDatasetNode.java
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HiveIncrPullSource.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/checkpointing/KafkaConnectHdfsProvider.java
{code}
> Audi all flink classes and route all FileSystem list and getStatus calls to
> metadata table
> --
>
> Key: HUDI-2201
> URL: https://issues.apache.org/jira/browse/HUDI-2201
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Common Core
>Reporter: sivabalan narayanan
>Priority: Major
>
>
> {code:java}
> grep -irl "fs.listStatus" hudi-*/* | grep -v Test
> hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
> hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
> {code}
>
> {code:java}
> grep -irl "fs.getFileStatus" hudi-*/* | grep -v Test
> hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkMergeHandle.java
> hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/profile/WriteProfiles.java
> hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
> hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
> {code}
> Some of them are valid ones which can't be routed. But just dumping all
> matches here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HUDI-1015) Audit all getAllPartitionPaths() calls and keep em out of fast path
[
https://issues.apache.org/jira/browse/HUDI-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384613#comment-17384613
]
sivabalan narayanan edited comment on HUDI-1015 at 7/21/21, 3:21 AM:
-
{code:java}
grep -irl "fs.listStatus" hudi-*/* | grep -v Test
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieLogFileCommand.java
hudi-cli/src/main/scala/org/apache/hudi/cli/DedupeSparkJob.scala
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/MarkerFiles.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/rollback/JavaListingBasedRollbackHelper.java
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FailSafeConsistencyGuard.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateAsyncOperations.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateDatasetNode.java
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HiveIncrPullSource.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/checkpointing/KafkaConnectHdfsProvider.java
{code}
All looks good here except for couple of flink classes. I am not sure if those
can't be routed via metadata table. Will create a sub-task.
was (Author: shivnarayan):
{code:java}
grep -irl "fs.listStatus" hudi-*/* | grep -v Test
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieLogFileCommand.java
hudi-cli/src/main/scala/org/apache/hudi/cli/DedupeSparkJob.scala
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/MarkerFiles.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/rollback/JavaListingBasedRollbackHelper.java
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FailSafeConsistencyGuard.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateAsyncOperations.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateDatasetNode.java
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HiveIncrPullSource.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/checkpointing/KafkaConnectHdfsProvider.java
{code}
All looks good here except for couple of flink classes. I am not sure if those
can't be routed via metadata table. Will create a sub-task.
> Audit all getAllPartitionPaths() calls and keep em out of fast path
> ---
>
> Key: HUDI-1015
>
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3307](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (5ad4271) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `44.92%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3307 +/- ## - Coverage 47.74% 2.81% -44.93% + Complexity 5591 85 -5506 Files 938 284 -654 Lines 41823 11881-29942 Branches 4213 986 -3227 - Hits 19968 335-19633 + Misses20070 11520 -8550 + Partials 1785 26 -1759 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.91%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[jira] [Created] (HUDI-2201) Audi all flink classes and route all FileSystem list and getStatus calls to metadata table
sivabalan narayanan created HUDI-2201:
-
Summary: Audi all flink classes and route all FileSystem list and
getStatus calls to metadata table
Key: HUDI-2201
URL: https://issues.apache.org/jira/browse/HUDI-2201
Project: Apache Hudi
Issue Type: Sub-task
Components: Common Core
Reporter: sivabalan narayanan
{code:java}
grep -irl "fs.listStatus" hudi-*/* | grep -v Test
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieLogFileCommand.java
hudi-cli/src/main/scala/org/apache/hudi/cli/DedupeSparkJob.scala
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/MarkerFiles.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/rollback/JavaListingBasedRollbackHelper.java
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FailSafeConsistencyGuard.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateAsyncOperations.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateDatasetNode.java
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HiveIncrPullSource.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/checkpointing/KafkaConnectHdfsProvider.java
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2139) MergeInto MOR Table May Result InCorrect Result
[
https://issues.apache.org/jira/browse/HUDI-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384618#comment-17384618
]
ASF GitHub Bot commented on HUDI-2139:
--
hudi-bot edited a comment on pull request #3230:
URL: https://github.com/apache/hudi/pull/3230#issuecomment-874605269
## CI report:
* 2568538186a80a3cea8470e01cafb654a825c75a UNKNOWN
* be436c40648b2a167c637c1ed98376a9cb15b16d Azure:
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1052)
* 2b6219dd1b9a223dcff9e37548a14e5e1dad9813 Azure:
[PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1065)
Bot commands
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MergeInto MOR Table May Result InCorrect Result
> ---
>
> Key: HUDI-2139
> URL: https://issues.apache.org/jira/browse/HUDI-2139
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Currently we process all the update-action and inert-action in the
> ExpressionPayload#
> getInsertValue without know whether the record is matched or not matched for
> MOR table. This may result in incorrect merge result. e.g.
> {code:java}
> Merge into h0
> using (select 2 as id, 'a1' as name, 10 as price from s) s0
> on h0.id = s0.id
> when matched then s0.id = 1 the update set id = s0.id, name = s0.name, price
> = 10
> when not matched then s0.id = 2 the insert (id,name,price) values(id,name,
> 20){code}
> If the id = 2 can matched the target table h0, but it cannot match the
> udpate-condition ( s0.id = 1), It should not update the table. However,
> currently we cannot know the matched state of the input record, it will goes
> to the not-matched actions and update the price to 20 finally. This is
> incorrect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HUDI-1015) Audit all getAllPartitionPaths() calls and keep em out of fast path
[
https://issues.apache.org/jira/browse/HUDI-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384616#comment-17384616
]
sivabalan narayanan edited comment on HUDI-1015 at 7/21/21, 3:17 AM:
-
{code:java}
grep -irl "fs.getFileStatus" hudi-*/* | grep -v Test
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkMergeHandle.java
hudi-common/src/main/java/org/apache/hudi/common/util/TablePathUtils.java
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
hudi-common/src/main/java/org/apache/hudi/common/fs/inline/InLineFileSystem.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FailSafeConsistencyGuard.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-common/src/main/java/org/apache/hudi/exception/TableNotFoundException.java
hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/profile/WriteProfiles.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
{code}
Again, not sure about flink classes. will file a sub-task.
Among rest of them, I could not find any calls that needs to be routed to
metadata table.
was (Author: shivnarayan):
{code:java}
grep -irl "fs.getFileStatus" hudi-*/* | grep -v Test
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkMergeHandle.java
hudi-common/src/main/java/org/apache/hudi/common/util/TablePathUtils.java
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
hudi-common/src/main/java/org/apache/hudi/common/fs/inline/InLineFileSystem.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FailSafeConsistencyGuard.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-common/src/main/java/org/apache/hudi/exception/TableNotFoundException.java
hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/profile/WriteProfiles.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
{code}
> Audit all getAllPartitionPaths() calls and keep em out of fast path
> ---
>
> Key: HUDI-1015
> URL: https://issues.apache.org/jira/browse/HUDI-1015
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Common Core, Writer Core
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Blocker
> Fix For: 0.9.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3230: [HUDI-2139] MergeInto MOR Table May Result InCorrect Result
hudi-bot edited a comment on pull request #3230: URL: https://github.com/apache/hudi/pull/3230#issuecomment-874605269 ## CI report: * 2568538186a80a3cea8470e01cafb654a825c75a UNKNOWN * be436c40648b2a167c637c1ed98376a9cb15b16d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1052) * 2b6219dd1b9a223dcff9e37548a14e5e1dad9813 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1065) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HUDI-1015) Audit all getAllPartitionPaths() calls and keep em out of fast path
[
https://issues.apache.org/jira/browse/HUDI-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384613#comment-17384613
]
sivabalan narayanan edited comment on HUDI-1015 at 7/21/21, 3:12 AM:
-
{code:java}
grep -irl "fs.listStatus" hudi-*/* | grep -v Test
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieLogFileCommand.java
hudi-cli/src/main/scala/org/apache/hudi/cli/DedupeSparkJob.scala
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/MarkerFiles.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/rollback/JavaListingBasedRollbackHelper.java
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FailSafeConsistencyGuard.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateAsyncOperations.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateDatasetNode.java
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HiveIncrPullSource.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/checkpointing/KafkaConnectHdfsProvider.java
{code}
All looks good here except for couple of flink classes. I am not sure if those
can't be routed via metadata table. Will create a sub-task.
was (Author: shivnarayan):
{{}}
{code:java}
grep -irl "fs.listStatus" hudi-*/* | grep -v Test
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieLogFileCommand.java
hudi-cli/src/main/scala/org/apache/hudi/cli/DedupeSparkJob.scala
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/MarkerFiles.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/rollback/JavaListingBasedRollbackHelper.java
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FailSafeConsistencyGuard.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateAsyncOperations.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateDatasetNode.java
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HiveIncrPullSource.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/checkpointing/KafkaConnectHdfsProvider.java
{code}
{{}}
> Audit all getAllPartitionPaths() calls and keep em out of fast path
> ---
>
> Key: HUDI-1015
> URL: https://issues.apache.org/jira/browse/HUDI-1015
> Project: Apache Hudi
> Issue Type:
[jira] [Commented] (HUDI-2177) Virtual keys support for Compaction
[ https://issues.apache.org/jira/browse/HUDI-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384617#comment-17384617 ] ASF GitHub Bot commented on HUDI-2177: -- hudi-bot edited a comment on pull request #3315: URL: https://github.com/apache/hudi/pull/3315#issuecomment-883851530 ## CI report: * b4b137968fc3ceed09408fdef934bba73764c5e5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1064) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virtual keys support for Compaction > --- > > Key: HUDI-2177 > URL: https://issues.apache.org/jira/browse/HUDI-2177 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virtual keys support for Compaction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3315: [HUDI-2177][HUDI-2200] Adding virtual keys support for MOR
hudi-bot edited a comment on pull request #3315: URL: https://github.com/apache/hudi/pull/3315#issuecomment-883851530 ## CI report: * b4b137968fc3ceed09408fdef934bba73764c5e5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1064) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2177) Virtual keys support for Compaction
[ https://issues.apache.org/jira/browse/HUDI-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384615#comment-17384615 ] ASF GitHub Bot commented on HUDI-2177: -- hudi-bot commented on pull request #3315: URL: https://github.com/apache/hudi/pull/3315#issuecomment-883851530 ## CI report: * b4b137968fc3ceed09408fdef934bba73764c5e5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virtual keys support for Compaction > --- > > Key: HUDI-2177 > URL: https://issues.apache.org/jira/browse/HUDI-2177 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virtual keys support for Compaction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1015) Audit all getAllPartitionPaths() calls and keep em out of fast path
[
https://issues.apache.org/jira/browse/HUDI-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384614#comment-17384614
]
sivabalan narayanan commented on HUDI-1015:
---
{code:java}
grep -irl "fs.listFiles" hudi-*/* | grep -v Test
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
{code}
> Audit all getAllPartitionPaths() calls and keep em out of fast path
> ---
>
> Key: HUDI-1015
> URL: https://issues.apache.org/jira/browse/HUDI-1015
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Common Core, Writer Core
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Blocker
> Fix For: 0.9.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-1015) Audit all getAllPartitionPaths() calls and keep em out of fast path
[
https://issues.apache.org/jira/browse/HUDI-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384616#comment-17384616
]
sivabalan narayanan commented on HUDI-1015:
---
{code:java}
grep -irl "fs.getFileStatus" hudi-*/* | grep -v Test
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkMergeHandle.java
hudi-common/src/main/java/org/apache/hudi/common/util/TablePathUtils.java
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
hudi-common/src/main/java/org/apache/hudi/common/fs/inline/InLineFileSystem.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FailSafeConsistencyGuard.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-common/src/main/java/org/apache/hudi/exception/TableNotFoundException.java
hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/profile/WriteProfiles.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
{code}
> Audit all getAllPartitionPaths() calls and keep em out of fast path
> ---
>
> Key: HUDI-1015
> URL: https://issues.apache.org/jira/browse/HUDI-1015
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Common Core, Writer Core
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Blocker
> Fix For: 0.9.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-1015) Audit all getAllPartitionPaths() calls and keep em out of fast path
[
https://issues.apache.org/jira/browse/HUDI-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384613#comment-17384613
]
sivabalan narayanan commented on HUDI-1015:
---
{{}}
{code:java}
grep -irl "fs.listStatus" hudi-*/* | grep -v Test
hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieLogFileCommand.java
hudi-cli/src/main/scala/org/apache/hudi/cli/DedupeSparkJob.scala
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/MarkerFiles.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/rollback/JavaListingBasedRollbackHelper.java
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackHelper.java
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FailSafeConsistencyGuard.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java
hudi-flink/src/main/java/org/apache/hudi/table/format/cow/CopyOnWriteInputFormat.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateAsyncOperations.java
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateDatasetNode.java
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HiveIncrPullSource.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/checkpointing/KafkaConnectHdfsProvider.java
{code}
{{}}
> Audit all getAllPartitionPaths() calls and keep em out of fast path
> ---
>
> Key: HUDI-1015
> URL: https://issues.apache.org/jira/browse/HUDI-1015
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Common Core, Writer Core
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Blocker
> Fix For: 0.9.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] hudi-bot commented on pull request #3315: [HUDI-2177][HUDI-2200] Adding virtual keys support for MOR
hudi-bot commented on pull request #3315: URL: https://github.com/apache/hudi/pull/3315#issuecomment-883851530 ## CI report: * b4b137968fc3ceed09408fdef934bba73764c5e5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1518) Remove replaced files logic from archival
[ https://issues.apache.org/jira/browse/HUDI-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384612#comment-17384612 ] ASF GitHub Bot commented on HUDI-1518: -- hudi-bot edited a comment on pull request #3310: URL: https://github.com/apache/hudi/pull/3310#issuecomment-883213429 ## CI report: * f0b369bdcadf55ee9bc58ad6bea618eee871b23f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1059) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove replaced files logic from archival > - > > Key: HUDI-1518 > URL: https://issues.apache.org/jira/browse/HUDI-1518 > Project: Apache Hudi > Issue Type: Task >Affects Versions: 0.9.0 >Reporter: satish >Priority: Minor > Labels: pull-request-available > Fix For: 0.9.0 > > > See > https://github.com/apache/hudi/blob/79ec7b4894b997183a6e10fdc19d34f5ab4ea437/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java#L324 > Archival tries to delete replaced files in 'deleteReplacedFileGroups' method. > We changed cleaner to delete replaced files. So this is no longer needed. We > can instead validate that replaced files no longer exist on disk during > archival. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3310: [HUDI-1518] Remove the logic that delete replaced file when archive
hudi-bot edited a comment on pull request #3310: URL: https://github.com/apache/hudi/pull/3310#issuecomment-883213429 ## CI report: * f0b369bdcadf55ee9bc58ad6bea618eee871b23f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1059) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2177) Virtual keys support for Compaction
[ https://issues.apache.org/jira/browse/HUDI-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384611#comment-17384611 ] ASF GitHub Bot commented on HUDI-2177: -- nsivabalan opened a new pull request #3315: URL: https://github.com/apache/hudi/pull/3315 ## What is the purpose of the pull request - Adding virtual keys support to MOR table - Compaction - Realtime read - Clustering Constraints: Only SimpleKeyGen is supported because during real time read (snapshot read), we can't afford to generated keys using complex key gens. Query times will shoot up and its unusable from a user standpoint. ## Brief change log - Introduced 2 additional configs to HoodieTableConfig to serialize the simple record field and simple partition field for the table. - Added virutal keys support to MOR table for compaction, clustering and realtime read. - Metadata table is ensured to work w/ virtual keyed MOR table. ## Verify this pull request *(Please pick either of the following options)* This change added tests and can be verified as follows: - Fixed TestHoodieMergeOnReadTable and TestHoodieBackedMetadata for virtual keys with MOR table. ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virtual keys support for Compaction > --- > > Key: HUDI-2177 > URL: https://issues.apache.org/jira/browse/HUDI-2177 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Fix For: 0.9.0 > > > Virtual keys support for Compaction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2200) Virtual keys support for MOR table: realtime read
sivabalan narayanan created HUDI-2200: - Summary: Virtual keys support for MOR table: realtime read Key: HUDI-2200 URL: https://issues.apache.org/jira/browse/HUDI-2200 Project: Apache Hudi Issue Type: Improvement Components: Writer Core Reporter: sivabalan narayanan -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2177) Virtual keys support for Compaction
[ https://issues.apache.org/jira/browse/HUDI-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2177: - Labels: pull-request-available (was: ) > Virtual keys support for Compaction > --- > > Key: HUDI-2177 > URL: https://issues.apache.org/jira/browse/HUDI-2177 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virtual keys support for Compaction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] nsivabalan opened a new pull request #3315: [HUDI-2177] Adding virtual keys support for MOR
nsivabalan opened a new pull request #3315: URL: https://github.com/apache/hudi/pull/3315 ## What is the purpose of the pull request - Adding virtual keys support to MOR table - Compaction - Realtime read - Clustering Constraints: Only SimpleKeyGen is supported because during real time read (snapshot read), we can't afford to generated keys using complex key gens. Query times will shoot up and its unusable from a user standpoint. ## Brief change log - Introduced 2 additional configs to HoodieTableConfig to serialize the simple record field and simple partition field for the table. - Added virutal keys support to MOR table for compaction, clustering and realtime read. - Metadata table is ensured to work w/ virtual keyed MOR table. ## Verify this pull request *(Please pick either of the following options)* This change added tests and can be verified as follows: - Fixed TestHoodieMergeOnReadTable and TestHoodieBackedMetadata for virtual keys with MOR table. ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2007) Adding spark bulk insert node
[ https://issues.apache.org/jira/browse/HUDI-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384606#comment-17384606 ] ASF GitHub Bot commented on HUDI-2007: -- codecov-commenter edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-860337193 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3074](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9c622e5) into [master](https://codecov.io/gh/apache/hudi/commit/634163a990569aa4463b58830396f455dd15340c?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (634163a) will **decrease** coverage by `44.92%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3074 +/- ## - Coverage 47.74% 2.81% -44.93% + Complexity 5592 85 -5507 Files 938 284 -654 Lines 41825 11881-29944 Branches 4213 986 -3227 - Hits 19971 335-19636 + Misses20071 11520 -8551 + Partials 1783 26 -1757 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.91%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
codecov-commenter edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-860337193 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3074](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9c622e5) into [master](https://codecov.io/gh/apache/hudi/commit/634163a990569aa4463b58830396f455dd15340c?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (634163a) will **decrease** coverage by `44.92%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3074 +/- ## - Coverage 47.74% 2.81% -44.93% + Complexity 5592 85 -5507 Files 938 284 -654 Lines 41825 11881-29944 Branches 4213 986 -3227 - Hits 19971 335-19636 + Misses20071 11520 -8551 + Partials 1783 26 -1757 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.91%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan
[ https://issues.apache.org/jira/browse/HUDI-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384604#comment-17384604 ] ASF GitHub Bot commented on HUDI-2194: -- hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 350f4266dd6951cb64ba492553e1099978f23fcf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1016) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1058) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Skip the latest N partitions when creating ClusteringPlan > - > > Key: HUDI-2194 > URL: https://issues.apache.org/jira/browse/HUDI-2194 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > As we known, SparkRecentDaysClusteringPlanStrategy is the default clustering > strategy to create ClusteringPlan. And it is useful when Hudi table is > partitioned by time. > > For now, users can set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` to control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > > |<- choose to cluster >| > Sometimes users also what to skip x partitions from latest when make > clustering plan because latest partitions contains lots of update data or > some reasons else. > > This patch will add a new config named ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` to set the > number of partitions to skip from latest when choosing partitions to create > ClusteringPlan > > for example users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 and > ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > |<- choose ->| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan
hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 350f4266dd6951cb64ba492553e1099978f23fcf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1016) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1058) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1518) Remove replaced files logic from archival
[ https://issues.apache.org/jira/browse/HUDI-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384602#comment-17384602 ] ASF GitHub Bot commented on HUDI-1518: -- codecov-commenter edited a comment on pull request #3310: URL: https://github.com/apache/hudi/pull/3310#issuecomment-883331907 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3310](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f0b369b) into [master](https://codecov.io/gh/apache/hudi/commit/572a2144124b90b2dbef8183ee19a95b21de2716?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (572a214) will **decrease** coverage by `45.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3310 +/- ## - Coverage 47.83% 2.82% -45.02% + Complexity 5565 85 -5480 Files 936 284 -652 Lines 41663 11868-29795 Branches 4197 985 -3212 - Hits 19929 335-19594 + Misses19960 11507 -8453 + Partials 1774 26 -1748 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.80%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/HoodieTimelineArchiveLog.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL0hvb2RpZVRpbWVsaW5lQXJjaGl2ZUxvZy5qYXZh) | `0.00% <ø> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3310: [HUDI-1518] Remove the logic that delete replaced file when archive
codecov-commenter edited a comment on pull request #3310: URL: https://github.com/apache/hudi/pull/3310#issuecomment-883331907 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3310](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f0b369b) into [master](https://codecov.io/gh/apache/hudi/commit/572a2144124b90b2dbef8183ee19a95b21de2716?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (572a214) will **decrease** coverage by `45.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3310 +/- ## - Coverage 47.83% 2.82% -45.02% + Complexity 5565 85 -5480 Files 936 284 -652 Lines 41663 11868-29795 Branches 4197 985 -3212 - Hits 19929 335-19594 + Misses19960 11507 -8453 + Partials 1774 26 -1748 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.80%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3310?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/table/HoodieTimelineArchiveLog.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL0hvb2RpZVRpbWVsaW5lQXJjaGl2ZUxvZy5qYXZh) | `0.00% <ø> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3310/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
hudi-bot edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883153186 ## CI report: * 915dccba0a3cf0fcaaf290dfb7f839cb44696fef Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1053) * 5ad4271d6c2cf000bc195d3945cbe2dd0f6cbb4a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1062) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
hudi-bot edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883153186 ## CI report: * 915dccba0a3cf0fcaaf290dfb7f839cb44696fef Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1053) * 5ad4271d6c2cf000bc195d3945cbe2dd0f6cbb4a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2139) MergeInto MOR Table May Result InCorrect Result
[
https://issues.apache.org/jira/browse/HUDI-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384597#comment-17384597
]
ASF GitHub Bot commented on HUDI-2139:
--
hudi-bot edited a comment on pull request #3230:
URL: https://github.com/apache/hudi/pull/3230#issuecomment-874605269
## CI report:
* 2568538186a80a3cea8470e01cafb654a825c75a UNKNOWN
* be436c40648b2a167c637c1ed98376a9cb15b16d Azure:
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1052)
* 2b6219dd1b9a223dcff9e37548a14e5e1dad9813 UNKNOWN
Bot commands
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MergeInto MOR Table May Result InCorrect Result
> ---
>
> Key: HUDI-2139
> URL: https://issues.apache.org/jira/browse/HUDI-2139
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Currently we process all the update-action and inert-action in the
> ExpressionPayload#
> getInsertValue without know whether the record is matched or not matched for
> MOR table. This may result in incorrect merge result. e.g.
> {code:java}
> Merge into h0
> using (select 2 as id, 'a1' as name, 10 as price from s) s0
> on h0.id = s0.id
> when matched then s0.id = 1 the update set id = s0.id, name = s0.name, price
> = 10
> when not matched then s0.id = 2 the insert (id,name,price) values(id,name,
> 20){code}
> If the id = 2 can matched the target table h0, but it cannot match the
> udpate-condition ( s0.id = 1), It should not update the table. However,
> currently we cannot know the matched state of the input record, it will goes
> to the not-matched actions and update the price to 20 finally. This is
> incorrect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3230: [HUDI-2139] MergeInto MOR Table May Result InCorrect Result
hudi-bot edited a comment on pull request #3230: URL: https://github.com/apache/hudi/pull/3230#issuecomment-874605269 ## CI report: * 2568538186a80a3cea8470e01cafb654a825c75a UNKNOWN * be436c40648b2a167c637c1ed98376a9cb15b16d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1052) * 2b6219dd1b9a223dcff9e37548a14e5e1dad9813 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2007) Adding spark bulk insert node
[ https://issues.apache.org/jira/browse/HUDI-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384593#comment-17384593 ] ASF GitHub Bot commented on HUDI-2007: -- hudi-bot edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-861653947 ## CI report: * 9c622e581cf38a62828f734514cf6204d902ac76 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1061) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1060) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Adding spark bulk insert node > - > > Key: HUDI-2007 > URL: https://issues.apache.org/jira/browse/HUDI-2007 > Project: Apache Hudi > Issue Type: Improvement > Components: Testing >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Adding spark bulk insert node and add config to enable row writing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
