[GitHub] [hudi] hudi-bot edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
hudi-bot edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871061118 ## CI report: * cf37ed3b99d63b48d06b7b14b8b6b200e52c6500 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=554) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3122: [HUDI-2048] HoodieRealtimeInputFormatUtils#groupLogsByBaseFile throws…
codecov-commenter edited a comment on pull request #3122: URL: https://github.com/apache/hudi/pull/3122#issuecomment-864855800 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3122?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3122](https://codecov.io/gh/apache/hudi/pull/3122?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (84bfc2d) into [master](https://codecov.io/gh/apache/hudi/commit/202887b8ca27eb6de808ba7a2e737b13ae9eb8c0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (202887b) will **decrease** coverage by `43.12%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3122?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3122 +/- ## - Coverage 46.19% 3.07% -43.13% + Complexity 5385 82 -5303 Files 921 280 -641 Lines 40040 10875-29165 Branches 42941106 -3188 - Hits 18495 334-18161 + Misses19661 10515 -9146 + Partials 1884 26 -1858 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.46% <ø> (-48.95%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3122?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3122/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3122/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3122/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3122/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3122/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[jira] [Assigned] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samrat Deb reassigned HUDI-1860: Assignee: Samrat Deb (was: Sagar Sumit) > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Samrat Deb >Priority: Major > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3122: [HUDI-2048] HoodieRealtimeInputFormatUtils#groupLogsByBaseFile throws…
hudi-bot edited a comment on pull request #3122: URL: https://github.com/apache/hudi/pull/3122#issuecomment-864851929 ## CI report: * 530787e9a14f8ac180399f4641a58e5eff6ada11 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=307) * 84bfc2d2c3bea3d088e3c1f8b0a390c586d80e48 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=555) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3122: [HUDI-2048] HoodieRealtimeInputFormatUtils#groupLogsByBaseFile throws…
hudi-bot edited a comment on pull request #3122: URL: https://github.com/apache/hudi/pull/3122#issuecomment-864851929 ## CI report: * 530787e9a14f8ac180399f4641a58e5eff6ada11 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=307) * 84bfc2d2c3bea3d088e3c1f8b0a390c586d80e48 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3122: [HUDI-2048] HoodieRealtimeInputFormatUtils#groupLogsByBaseFile throws…
danny0405 commented on pull request #3122: URL: https://github.com/apache/hudi/pull/3122#issuecomment-871114694 @vinothchandar I have modified to support pure logs file group reading, can you take a look again, thanks ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1468) incremental read support with clustering
[ https://issues.apache.org/jira/browse/HUDI-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371819#comment-17371819 ] Vinoth Chandar commented on HUDI-1468: -- [~309637554] do you plan to work on this? It would be good to have this in the next release. > incremental read support with clustering > > > Key: HUDI-1468 > URL: https://issues.apache.org/jira/browse/HUDI-1468 > Project: Apache Hudi > Issue Type: Sub-task > Components: Incremental Pull >Affects Versions: 0.9.0 >Reporter: satish >Assignee: liwei >Priority: Blocker > Fix For: 0.9.0 > > > As part of clustering, metadata such as hoodie_commit_time changes for > records that are clustered. This is specific to > SparkBulkInsertBasedRunClusteringStrategy implementation. Figure out a way to > carry commit_time from original record to support incremental queries. > Also, incremental queries dont work with 'replacecommit' used by clustering > HUDI-1264. Change incremental query to work for replacecommits created by > Clustering. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] wangxianghu commented on a change in pull request #3170: [HUDI-1930] Bootstrap support configure KeyGenerator by type
wangxianghu commented on a change in pull request #3170: URL: https://github.com/apache/hudi/pull/3170#discussion_r661144248 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java ## @@ -353,14 +353,20 @@ private static int deduplicatePartitionPath(JavaSparkContext jsc, String duplica private static int doBootstrap(JavaSparkContext jsc, String tableName, String tableType, String basePath, Review comment: yes, filed another ticket to rack this : https://issues.apache.org/jira/browse/HUDI-2104 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-2104) Add Builder for bootstrap cli args
Xianghu Wang created HUDI-2104: -- Summary: Add Builder for bootstrap cli args Key: HUDI-2104 URL: https://issues.apache.org/jira/browse/HUDI-2104 Project: Apache Hudi Issue Type: Task Components: CLI Reporter: Xianghu Wang Assignee: Xianghu Wang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
codecov-commenter edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871069429 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
codecov-commenter edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871069429 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3186](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cf37ed3) into [master](https://codecov.io/gh/apache/hudi/commit/202887b8ca27eb6de808ba7a2e737b13ae9eb8c0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (202887b) will **decrease** coverage by `17.70%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3186 +/- ## = - Coverage 46.19% 28.48% -17.71% + Complexity 5385 1265 -4120 = Files 921 376 -545 Lines 4004014327-25713 Branches 4294 1459 -2835 = - Hits 18495 4081-14414 + Misses19661 9951 -9710 + Partials 1884 295 -1589 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.16%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.47% <50.00%> (-48.58%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.44% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ache/hudi/hive/HiveMetastoreBasedLockProvider.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZU1ldGFzdG9yZUJhc2VkTG9ja1Byb3ZpZGVyLmphdmE=) | `60.63% <50.00%> (+0.42%)` | :arrow_up: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
codecov-commenter edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871069429 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3186](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cf37ed3) into [master](https://codecov.io/gh/apache/hudi/commit/202887b8ca27eb6de808ba7a2e737b13ae9eb8c0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (202887b) will **decrease** coverage by `29.88%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3186 +/- ## = - Coverage 46.19% 16.31% -29.89% + Complexity 5385 477 -4908 = Files 921 280 -641 Lines 4004010876-29164 Branches 4294 1107 -3187 = - Hits 18495 1774-16721 + Misses19661 8944-10717 + Partials 1884 158 -1726 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.47% <50.00%> (-48.58%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.44% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ache/hudi/hive/HiveMetastoreBasedLockProvider.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZU1ldGFzdG9yZUJhc2VkTG9ja1Byb3ZpZGVyLmphdmE=) | `60.63% <50.00%> (+0.42%)` | :arrow_up: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
hudi-bot edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871061118 ## CI report: * cf3420f154c8f13fefe4ff4ea589322ac50249b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=553) * cf37ed3b99d63b48d06b7b14b8b6b200e52c6500 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=554) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch asf-site updated (d544e97 -> c18712c)
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git. from d544e97 [MINOR] [HUDI-1577] Add caveat for multiwriter usage (#3165) add c18712c Travis CI build asf-site No new revisions were added by this update. Summary of changes: content/docs/concurrency_control.html | 6 ++ 1 file changed, 6 insertions(+)
[GitHub] [hudi] vinothchandar merged pull request #3165: [MINOR] Add caveat for multiwriter usage
vinothchandar merged pull request #3165: URL: https://github.com/apache/hudi/pull/3165 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch asf-site updated: [MINOR] [HUDI-1577] Add caveat for multiwriter usage (#3165)
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new d544e97 [MINOR] [HUDI-1577] Add caveat for multiwriter usage (#3165) d544e97 is described below commit d544e97ef14f3b7e540f335f3d3b3833a4977e01 Author: n3nash AuthorDate: Tue Jun 29 21:52:18 2021 -0700 [MINOR] [HUDI-1577] Add caveat for multiwriter usage (#3165) --- docs/_docs/2_9_concurrency_control.md | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/_docs/2_9_concurrency_control.md b/docs/_docs/2_9_concurrency_control.md index 918556e..4ae66ed 100644 --- a/docs/_docs/2_9_concurrency_control.md +++ b/docs/_docs/2_9_concurrency_control.md @@ -142,4 +142,9 @@ Remove the following settings that were used to enable multi-writer or override ``` hoodie.write.concurrency.mode=single_writer hoodie.cleaner.policy.failed.writes=EAGER -``` \ No newline at end of file +``` + +## Caveats + +If you are using the `WriteClient` API, please note that multiple writes to the table need to be initiated from 2 different instances of the write client. +It is NOT recommended to use the same instance of the write client to perform multi writing. \ No newline at end of file
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
codecov-commenter edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871069429 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3186](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cf37ed3) into [master](https://codecov.io/gh/apache/hudi/commit/202887b8ca27eb6de808ba7a2e737b13ae9eb8c0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (202887b) will **decrease** coverage by `43.11%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3186 +/- ## - Coverage 46.19% 3.08% -43.12% + Complexity 5385 85 -5300 Files 921 280 -641 Lines 40040 10876-29164 Branches 42941107 -3187 - Hits 18495 335-18160 + Misses19661 10515 -9146 + Partials 1884 26 -1858 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.47% <50.00%> (-48.58%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.46% <ø> (-48.95%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ache/hudi/hive/HiveMetastoreBasedLockProvider.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZU1ldGFzdG9yZUJhc2VkTG9ja1Byb3ZpZGVyLmphdmE=) | `60.63% <50.00%> (+0.42%)` | :arrow_up: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
hudi-bot edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871061118 ## CI report: * cf3420f154c8f13fefe4ff4ea589322ac50249b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=553) * cf37ed3b99d63b48d06b7b14b8b6b200e52c6500 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash edited a comment on pull request #3183: [HUDI-2090]when hudi metadata is enabled, use different users to quer…
n3nash edited a comment on pull request #3183: URL: https://github.com/apache/hudi/pull/3183#issuecomment-871080919 @nsivabalan I ran into this today while testing :) @xiarixiaoyao The question is why is the base path "/tmp/view_map". It should ideally be just "/tmp". I see that this is coming from here -> https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewStorageConfig.java#L50 and the change was made after discussions here -> https://github.com/apache/hudi/issues/768. @garyli1019 Any specific reason we changed to "view_map" ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash commented on pull request #3165: [MINOR] Add caveat for multiwriter usage
n3nash commented on pull request #3165: URL: https://github.com/apache/hudi/pull/3165#issuecomment-871081550 @nsivabalan Can you review this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
hudi-bot edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871061118 ## CI report: * cf3420f154c8f13fefe4ff4ea589322ac50249b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=553) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash commented on pull request #3183: [HUDI-2090]when hudi metadata is enabled, use different users to quer…
n3nash commented on pull request #3183: URL: https://github.com/apache/hudi/pull/3183#issuecomment-871080919 @nsivabalan I ran into this today while testing :) @xiarixiaoyao The question is why is the base path "/tmp/view_map". It should ideally be just "/tmp". Can you look into this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
hudi-bot edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871061118 ## CI report: * 4fa699a875133e4838700320cce2bc2fe6f738a2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=551) * cf3420f154c8f13fefe4ff4ea589322ac50249b0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=553) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
codecov-commenter edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871069429 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3186](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cf3420f) into [master](https://codecov.io/gh/apache/hudi/commit/202887b8ca27eb6de808ba7a2e737b13ae9eb8c0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (202887b) will **increase** coverage by `12.91%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3186 +/- ## = + Coverage 46.19% 59.10% +12.91% + Complexity 5385 1189 -4196 = Files 921 163 -758 Lines 40040 6389-33651 Branches 4294 670 -3624 = - Hits 18495 3776-14719 + Misses19661 2341-17320 + Partials 1884 272 -1612 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `66.85% <ø> (+36.39%)` | :arrow_up: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `50.00% <ø> (-8.41%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.63%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...org/apache/hudi/utilities/HDFSParquetImporter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hERlNQYXJxdWV0SW1wb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-71.82%)` | :arrow_down: | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
codecov-commenter edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871069429 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3186](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (4fa699a) into [master](https://codecov.io/gh/apache/hudi/commit/202887b8ca27eb6de808ba7a2e737b13ae9eb8c0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (202887b) will **increase** coverage by `12.89%`. > The diff coverage is `n/a`. > :exclamation: Current head 4fa699a differs from pull request most recent head cf3420f. Consider uploading reports for the commit cf3420f to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3186 +/- ## = + Coverage 46.19% 59.08% +12.89% + Complexity 5385 1188 -4197 = Files 921 163 -758 Lines 40040 6389-33651 Branches 4294 670 -3624 = - Hits 18495 3775-14720 + Misses19661 2341-17320 + Partials 1884 273 -1611 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `66.85% <ø> (+36.39%)` | :arrow_up: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `49.96% <ø> (-8.45%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.63%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...org/apache/hudi/utilities/HDFSParquetImporter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hERlNQYXJxdWV0SW1wb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-71.82%)` | :arrow_down: | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
codecov-commenter edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871069429 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3186](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cf3420f) into [master](https://codecov.io/gh/apache/hudi/commit/202887b8ca27eb6de808ba7a2e737b13ae9eb8c0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (202887b) will **increase** coverage by `3.80%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3186 +/- ## + Coverage 46.19% 50.00% +3.80% + Complexity 5385 401-4984 Files 921 67 -854 Lines 40040 2938 -37102 Branches 4294 318-3976 - Hits 18495 1469 -17026 + Misses19661 1334 -18327 + Partials 1884 135-1749 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `?` | | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `50.00% <ø> (-8.41%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.63%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...org/apache/hudi/utilities/HDFSParquetImporter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hERlNQYXJxdWV0SW1wb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-71.82%)` | :arrow_down: | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | :arrow_down: | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=)
[GitHub] [hudi] yanghua commented on a change in pull request #3170: [HUDI-1930] Bootstrap support configure KeyGenerator by type
yanghua commented on a change in pull request #3170: URL: https://github.com/apache/hudi/pull/3170#discussion_r661106077 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java ## @@ -353,14 +353,20 @@ private static int deduplicatePartitionPath(JavaSparkContext jsc, String duplica private static int doBootstrap(JavaSparkContext jsc, String tableName, String tableType, String basePath, Review comment: The original parameter list is not very graceful. I suggest we can only receive `args`, then use a builder to build(extract) the metadata of the bootstrap action. wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
codecov-commenter commented on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871069429 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3186](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (4fa699a) into [master](https://codecov.io/gh/apache/hudi/commit/202887b8ca27eb6de808ba7a2e737b13ae9eb8c0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (202887b) will **increase** coverage by `3.77%`. > The diff coverage is `n/a`. > :exclamation: Current head 4fa699a differs from pull request most recent head cf3420f. Consider uploading reports for the commit cf3420f to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3186 +/- ## + Coverage 46.19% 49.96% +3.77% + Complexity 5385 400-4985 Files 921 67 -854 Lines 40040 2938 -37102 Branches 4294 318-3976 - Hits 18495 1468 -17027 + Misses19661 1334 -18327 + Partials 1884 136-1748 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `?` | | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `49.96% <ø> (-8.45%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3186?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.63%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...org/apache/hudi/utilities/HDFSParquetImporter.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hERlNQYXJxdWV0SW1wb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-71.82%)` | :arrow_down: | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/3186/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3182: [HUDI-2089]fix the bug that metatable cannot support non_partition table
codecov-commenter edited a comment on pull request #3182: URL: https://github.com/apache/hudi/pull/3182#issuecomment-870529394 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
hudi-bot edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871061118 ## CI report: * 4fa699a875133e4838700320cce2bc2fe6f738a2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=551) * cf3420f154c8f13fefe4ff4ea589322ac50249b0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3182: [HUDI-2089]fix the bug that metatable cannot support non_partition table
codecov-commenter edited a comment on pull request #3182: URL: https://github.com/apache/hudi/pull/3182#issuecomment-870529394 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3182](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9582f7d) into [master](https://codecov.io/gh/apache/hudi/commit/0749cc826a9ef6b04e2063074b7abac0e69414a0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0749cc8) will **decrease** coverage by `15.77%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3182 +/- ## = - Coverage 44.25% 28.47% -15.78% + Complexity 4595 1262 - = Files 825 376 -449 Lines 3657714327-22250 Branches 3942 1459 -2483 = - Hits 16187 4080-12107 + Misses18642 9952 -8690 + Partials 1748 295 -1453 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <0.00%> (+5.84%)` | :arrow_up: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.44% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] liijiankang opened a new issue #3187: [SUPPORT]Unable to shutdown local metastore client
liijiankang opened a new issue #3187: URL: https://github.com/apache/hudi/issues/3187 **Describe the problem you faced** I used the code on the hudi master branch to write data into hudi and synced it to hive, and encountered this exception ："ERROR hive.metastore: Unable to shutdown local metastore client" . I'm not sure if this is a hive problem or a hudi problem. I found this link in the hive community. “https://issues.apache.org/jira/browse/HIVE-2883”. I did not encounter this problem when using the 0.6.0 version of Hudi. **Stacktrace** ```  ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
hudi-bot edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871061118 ## CI report: * 4fa699a875133e4838700320cce2bc2fe6f738a2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=551) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3182: [HUDI-2089]fix the bug that metatable cannot support non_partition table
hudi-bot edited a comment on pull request #3182: URL: https://github.com/apache/hudi/pull/3182#issuecomment-870381931 ## CI report: * 9582f7dcde72e98c09fc12e0f69f5beef28dbaea Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=550) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
hudi-bot edited a comment on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871061118 ## CI report: * 4fa699a875133e4838700320cce2bc2fe6f738a2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=551) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #3073: [HUDI-2006] Adding more yaml templates to test suite
nsivabalan merged pull request #3073: URL: https://github.com/apache/hudi/pull/3073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated (202887b -> 5564c7e)
This is an automated email from the ASF dual-hosted git repository.
sivabalan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git.
from 202887b [HUDI-2092] Fix NPE caused by
FlinkStreamerConfig#writePartitionUrlEncode null value (#3176)
add 5564c7e [HUDI-2006] Adding more yaml templates to test suite (#3073)
No new revisions were added by this update.
Summary of changes:
.../test-suite/templates/clustering.yaml.template | 2 +
...yaml.template => long_test_suite.yaml.template} | 7 +-
...ml.template => medium_test_suite.yaml.template} | 9 +-
...-running.yaml.template => sanity.yaml.template} | 14 +--
docker/generate_test_suite.sh | 136 +
hudi-integ-test/README.md | 30 +
6 files changed, 157 insertions(+), 41 deletions(-)
copy docker/demo/config/test-suite/templates/{long-running.yaml.template =>
long_test_suite.yaml.template} (90%)
copy docker/demo/config/test-suite/templates/{long-running.yaml.template =>
medium_test_suite.yaml.template} (90%)
rename docker/demo/config/test-suite/templates/{long-running.yaml.template =>
sanity.yaml.template} (88%)
[GitHub] [hudi] hudi-bot commented on pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
hudi-bot commented on pull request #3186: URL: https://github.com/apache/hudi/pull/3186#issuecomment-871061118 ## CI report: * 4fa699a875133e4838700320cce2bc2fe6f738a2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2099) hive lock which state is WATING should be released, otherwise this hive lock will be locked forever
[ https://issues.apache.org/jira/browse/HUDI-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2099: - Labels: pull-request-available (was: ) > hive lock which state is WATING should be released, otherwise this hive > lock will be locked forever > - > > Key: HUDI-2099 > URL: https://issues.apache.org/jira/browse/HUDI-2099 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Affects Versions: 0.8.0 > Environment: spark3.1.1 > hive3.1.1 > hadoop3.1.1 >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.8.0 > > > when we acquire hive lock failed and the lock state is WATING， we should > release this WATING lock； otherwise this hive lock will be locked forever。 > test step： > use hive lock to control concurrent write for hudi， let‘s call this lock > hive_lock > start three writers to write hudi table by using hive_lock concurrently， one > of the writer will failed to acquire hive lock due to competition issues。 > *Exception in thread "main" org.apache.hudi.exception.HoodieLockException: > Unable to acquire lock, lock object LockResponse(lockid:76, state:WAITING)* > > start another writer to write hudi table by using same hive_lock， then we > find hive_lock is locked forever, we have no way to acquire it > *Exception in thread "main" org.apache.hudi.exception.HoodieLockException: > Unable to acquire lock, lock object LockResponse(lockid:87, state:WAITING)* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] xiarixiaoyao opened a new pull request #3186: [HUDI-2099]hive lock which state is WATING should be released, otherw…
xiarixiaoyao opened a new pull request #3186: URL: https://github.com/apache/hudi/pull/3186 …ise this hive lock will be locked forever ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request when we acquire hive lock failed and the lock state is WATING， we should release this WATING lock； otherwise this hive lock will be locked forever。 test step： use hive lock to control concurrent write for hudi， let‘s call this lock hive_lock start three writers to write hudi table by using hive_lock concurrently， one of the writer will failed to acquire hive lock due to competition issues。 **Exception in thread "main" org.apache.hudi.exception.HoodieLockException: Unable to acquire lock, lock object LockResponse(lockid:76, state:WAITING)** start another writer to write hudi table by using same hive_lock， then we find hive_lock is locked forever, we have no way to acquire it **Exception in thread "main" org.apache.hudi.exception.HoodieLockException: Unable to acquire lock, lock object LockResponse(lockid:87, state:WAITING)** ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request new UT added。 ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3182: [HUDI-2089]fix the bug that metatable cannot support non_partition table
codecov-commenter edited a comment on pull request #3182: URL: https://github.com/apache/hudi/pull/3182#issuecomment-870529394 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3182](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9582f7d) into [master](https://codecov.io/gh/apache/hudi/commit/0749cc826a9ef6b04e2063074b7abac0e69414a0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0749cc8) will **decrease** coverage by `27.95%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3182 +/- ## = - Coverage 44.25% 16.30% -27.96% + Complexity 4595 474 -4121 = Files 825 280 -545 Lines 3657710876-25701 Branches 3942 1107 -2835 = - Hits 16187 1773-14414 + Misses18642 8945 -9697 + Partials 1748 158 -1590 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-16.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.44% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] garyli1019 closed pull request #2378: [HUDI-1491] Support partition pruning for MOR snapshot query
garyli1019 closed pull request #2378: URL: https://github.com/apache/hudi/pull/2378 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] garyli1019 commented on pull request #2378: [HUDI-1491] Support partition pruning for MOR snapshot query
garyli1019 commented on pull request #2378: URL: https://github.com/apache/hudi/pull/2378#issuecomment-871055741 @yui2010 thanks for your contribution, I think appending the partition column to the data source table should be the ultimate way. closing this pr for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3182: [HUDI-2089]fix the bug that metatable cannot support non_partition table
codecov-commenter edited a comment on pull request #3182: URL: https://github.com/apache/hudi/pull/3182#issuecomment-870529394 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3182](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9582f7d) into [master](https://codecov.io/gh/apache/hudi/commit/0749cc826a9ef6b04e2063074b7abac0e69414a0?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0749cc8) will **decrease** coverage by `41.18%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3182 +/- ## - Coverage 44.25% 3.07% -41.19% + Complexity 4595 82 -4513 Files 825 280 -545 Lines 36577 10876-25701 Branches 39421107 -2835 - Hits 16187 334-15853 + Misses18642 10516 -8126 + Partials 1748 26 -1722 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-16.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.46% <ø> (-48.95%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3182?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3182/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] yanghua commented on a change in pull request #3177: [HUDI-2093] Fix empty avro schema path caused by duplicate parameters
yanghua commented on a change in pull request #3177: URL: https://github.com/apache/hudi/pull/3177#discussion_r661085373 ## File path: hudi-flink/src/main/java/org/apache/hudi/streamer/FlinkStreamerConfig.java ## @@ -287,7 +283,6 @@ org.apache.flink.configuration.Configuration conf = fromMap(propsMap); conf.setString(FlinkOptions.PATH, config.targetBasePath); -conf.setString(FlinkOptions.READ_AVRO_SCHEMA_PATH, config.readSchemaFilePath); Review comment: Can we also refactor the `READ_AVRO_SCHEMA_PATH ` config option. I would like to remove `READ_`. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3182: [HUDI-2089]fix the bug that metatable cannot support non_partition table
hudi-bot edited a comment on pull request #3182: URL: https://github.com/apache/hudi/pull/3182#issuecomment-870381931 ## CI report: * d59d9848553983cedbad7e2a08d4ac33b725a68a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=531) * 9582f7dcde72e98c09fc12e0f69f5beef28dbaea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=550) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3182: [HUDI-2089]fix the bug that metatable cannot support non_partition table
hudi-bot edited a comment on pull request #3182: URL: https://github.com/apache/hudi/pull/3182#issuecomment-870381931 ## CI report: * d59d9848553983cedbad7e2a08d4ac33b725a68a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=531) * 9582f7dcde72e98c09fc12e0f69f5beef28dbaea UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3183: [HUDI-2090]when hudi metadata is enabled, use different users to quer…
xiarixiaoyao commented on a change in pull request #3183:
URL: https://github.com/apache/hudi/pull/3183#discussion_r661075620
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java
##
@@ -111,8 +111,14 @@ private void initFile(File writeOnlyFile) throws
IOException {
if (writeOnlyFile.exists()) {
writeOnlyFile.delete();
}
-if (!writeOnlyFile.getParentFile().exists()) {
- writeOnlyFile.getParentFile().mkdir();
+File parentDir = writeOnlyFile.getParentFile();
+if (!parentDir.exists()) {
+ parentDir.mkdir();
+ // we should set all right for other user.
Review comment:
@leesf sorry, i think it's hard to simulate different users, since we
have no right to create a new user in linux（Linux permission problem）. hope you
can give me some advice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3183: [HUDI-2090]when hudi metadata is enabled, use different users to quer…
xiarixiaoyao commented on a change in pull request #3183:
URL: https://github.com/apache/hudi/pull/3183#discussion_r661075620
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java
##
@@ -111,8 +111,14 @@ private void initFile(File writeOnlyFile) throws
IOException {
if (writeOnlyFile.exists()) {
writeOnlyFile.delete();
}
-if (!writeOnlyFile.getParentFile().exists()) {
- writeOnlyFile.getParentFile().mkdir();
+File parentDir = writeOnlyFile.getParentFile();
+if (!parentDir.exists()) {
+ parentDir.mkdir();
+ // we should set all right for other user.
Review comment:
@leesf sorry, i think it's hard to simulate different users, since we
have no right to create a new user in linux（authority issues）. hope you can
give me some advice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3139: [HUDI-2058]support incremental query for insert_overwrite_table/insert_overwrite operation on cow table
xiarixiaoyao edited a comment on pull request #3139: URL: https://github.com/apache/hudi/pull/3139#issuecomment-868180580 @leesf @nsivabalan @garyli1019 could you help me review this pr, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3181: [HUDI-2059] When log exists in mor table, clustering is triggered. The query result shows that the update record in log is los
xiarixiaoyao commented on a change in pull request #3181:
URL: https://github.com/apache/hudi/pull/3181#discussion_r661072382
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieFileSliceReader.java
##
@@ -41,11 +43,26 @@
while (baseIterator.hasNext()) {
GenericRecord record = (GenericRecord) baseIterator.next();
HoodieRecord hoodieRecord =
SpillableMapUtils.convertToHoodieRecordPayload(record, payloadClass);
- scanner.processNextRecord(hoodieRecord);
+ processNextRecord(scanner, hoodieRecord);
}
return new HoodieFileSliceReader(scanner.iterator());
}
+ private static void processNextRecord(HoodieMergedLogRecordScanner scanner,
HoodieRecord hoodieRecord) {
Review comment:
thanks for your review.commit3 is only to use trigger clustering.
in the commit1 , we create a table and the age column is assigned to 1
in the commit2，update age value to 1001 where keyid < 5 to produce log files
in the commit3，only use to trigger clustering and find the bug。
In the previous code， we combine base file record and log file record by
call HoodieMergedLogRecordeScanner.processNextRecord, however this function is
not suitable for this scene，if we call this function，the record from base file
will be kept， and the record from log file will be discard，this is wrong。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
codecov-commenter edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-864499795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
codecov-commenter edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-864499795 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3117](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (72a3594) into [master](https://codecov.io/gh/apache/hudi/commit/c08fbb4268ee4b227452fd27d5e6ba322eeef00e?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (c08fbb4) will **decrease** coverage by `27.82%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3117 +/- ## = - Coverage 46.01% 18.19% -27.83% + Complexity 5306 535 -4771 = Files 911 286 -625 Lines 3947611330-28146 Branches 4254 1131 -3123 = - Hits 18166 2061-16105 + Misses19456 9091-10365 + Partials 1854 178 -1676 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-46.35%)` | :arrow_down: | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `58.40% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-17.01%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
hudi-bot edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-864499549 ## CI report: * 72a359459c780ae3d5f0e6b598aa363513ca28ca Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=548) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3183: [HUDI-2090]when hudi metadata is enabled, use different users to quer…
xiarixiaoyao edited a comment on pull request #3183: URL: https://github.com/apache/hudi/pull/3183#issuecomment-871031204 @garyli1019 thanks for you review。 not two users query the same DiskBasedMap。all users share the same directory /tmp/view_map for query. think that, using user1 to query hudi_table with sparksql, then a diskBasedMap instance will be create and this diskBasedMap will create /tmp/view_map directory at same time. notice the permission of view_maps belongs to user1. then using user2 to query hudi_table with hive, a new diskBasedMap instance will create, and this diskBasedMap want to create tempfile in /tmp/view_map directory for query, however it has no permisson, since view_map is create by user1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3183: [HUDI-2090]when hudi metadata is enabled, use different users to quer…
xiarixiaoyao commented on a change in pull request #3183:
URL: https://github.com/apache/hudi/pull/3183#discussion_r661064156
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java
##
@@ -111,8 +111,14 @@ private void initFile(File writeOnlyFile) throws
IOException {
if (writeOnlyFile.exists()) {
writeOnlyFile.delete();
}
-if (!writeOnlyFile.getParentFile().exists()) {
- writeOnlyFile.getParentFile().mkdir();
+File parentDir = writeOnlyFile.getParentFile();
+if (!parentDir.exists()) {
+ parentDir.mkdir();
+ // we should set all right for other user.
Review comment:
thanks for you reivew. i will try to add UT。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3183: [HUDI-2090]when hudi metadata is enabled, use different users to quer…
xiarixiaoyao commented on pull request #3183: URL: https://github.com/apache/hudi/pull/3183#issuecomment-871031204 @garyli1019 thanks for you review。 not two users query the same DiskBasedMap。 think that, using user1 to query hudi_table with sparksql, then a diskBasedMap instance will be create and this diskBasedMap will create /tmp/view_map directory at same time. notice the permission of view_maps belongs to user1 then using user2 to query hudi_table with hive, a new diskBasedMap instance will create, and this diskBasedMap want to create tempfile in /tmp/view_map directory for query, however it has no permisson, since view_map is create by user1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3183: [HUDI-2090]when hudi metadata is enabled, use different users to quer…
xiarixiaoyao commented on pull request #3183: URL: https://github.com/apache/hudi/pull/3183#issuecomment-871026878 @nsivabalan It's easy to reproduce， pls try it thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2092) Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value
[ https://issues.apache.org/jira/browse/HUDI-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2092: --- Fix Version/s: 0.9.0 > Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value > > > Key: HUDI-2092 > URL: https://issues.apache.org/jira/browse/HUDI-2092 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2092) Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value
[ https://issues.apache.org/jira/browse/HUDI-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2092. -- Resolution: Fixed 202887b8ca27eb6de808ba7a2e737b13ae9eb8c0 > Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value > > > Key: HUDI-2092 > URL: https://issues.apache.org/jira/browse/HUDI-2092 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[hudi] branch master updated (f665db0 -> 202887b)
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from f665db0 [HUDI-2085] Support specify compaction paralleism and compaction target io for flink batch compaction (#3169) add 202887b [HUDI-2092] Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value (#3176) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/hudi/streamer/FlinkStreamerConfig.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [hudi] yanghua merged pull request #3176: [HUDI-2092] Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEn…
yanghua merged pull request #3176: URL: https://github.com/apache/hudi/pull/3176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HUDI-1983) Add key gen type config to Configurations page in hudi website
[ https://issues.apache.org/jira/browse/HUDI-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianghu Wang resolved HUDI-1983. Resolution: Resolved Resolved via asf-site branch : 0507aef156c6aada4ccfa3d760da1a0b58c5a160 > Add key gen type config to Configurations page in hudi website > -- > > Key: HUDI-1983 > URL: https://issues.apache.org/jira/browse/HUDI-1983 > Project: Apache Hudi > Issue Type: Task >Reporter: sivabalan narayanan >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > > Add key gen type config to Configurations page in hudi website -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
codecov-commenter edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-864499795 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3117](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (72a3594) into [master](https://codecov.io/gh/apache/hudi/commit/c08fbb4268ee4b227452fd27d5e6ba322eeef00e?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (c08fbb4) will **decrease** coverage by `29.73%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3117 +/- ## = - Coverage 46.01% 16.28% -29.74% + Complexity 5306 473 -4833 = Files 911 280 -631 Lines 3947610881-28595 Branches 4254 1107 -3147 = - Hits 18166 1772-16394 + Misses19456 8950-10506 + Partials 1854 159 -1695 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-46.35%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.40% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-17.01%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3183: [HUDI-2090]when hudi metadata is enabled, use different users to quer…
xiarixiaoyao commented on a change in pull request #3183:
URL: https://github.com/apache/hudi/pull/3183#discussion_r661064156
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java
##
@@ -111,8 +111,14 @@ private void initFile(File writeOnlyFile) throws
IOException {
if (writeOnlyFile.exists()) {
writeOnlyFile.delete();
}
-if (!writeOnlyFile.getParentFile().exists()) {
- writeOnlyFile.getParentFile().mkdir();
+File parentDir = writeOnlyFile.getParentFile();
+if (!parentDir.exists()) {
+ parentDir.mkdir();
+ // we should set all right for other user.
Review comment:
ok， i will try to add UT。 thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3182: [HUDI-2089]fix the bug that metatable cannot support non_partition table
xiarixiaoyao commented on a change in pull request #3182:
URL: https://github.com/apache/hudi/pull/3182#discussion_r661062340
##
File path:
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala
##
@@ -677,4 +677,40 @@ class TestMORDataSource extends HoodieClientTestBase {
assertEquals(partitionCounts("2021/03/03"), count7)
}
+
+ @Test
+ def testNonPartitionTableWithMetaTableEnabled(): Unit = {
+val df = spark.range(0, 10).toDF("keyid")
+ .withColumn("col3", expr("keyid"))
+ .withColumn("age", expr("keyid + 1000"))
+
+df.write.format("hudi").
+ option(DataSourceWriteOptions.TABLE_TYPE_OPT_KEY,
DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL).
Review comment:
yes， thanks。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3182: [HUDI-2089]fix the bug that metatable cannot support non_partition table
xiarixiaoyao commented on a change in pull request #3182:
URL: https://github.com/apache/hudi/pull/3182#discussion_r661062254
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##
@@ -376,7 +381,9 @@ private void bootstrapFromFilesystem(HoodieEngineContext
engineContext, HoodieTa
if (p.getRight().length > filesInDir.size()) {
// Is a partition. Add all data files to result.
Review comment:
thanks， i will do it。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3182: [HUDI-2089]fix the bug that metatable cannot support non_partition table
xiarixiaoyao commented on pull request #3182: URL: https://github.com/apache/hudi/pull/3182#issuecomment-871024198 @nsivabalan thanks， i will do it。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3073: [HUDI-2006] Adding more yaml templates to test suite
codecov-commenter edited a comment on pull request #3073: URL: https://github.com/apache/hudi/pull/3073#issuecomment-860281795 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3073](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6d4ffe0) into [master](https://codecov.io/gh/apache/hudi/commit/0d0dc6fb07e0c5496224c75052ab4f43d57b40f6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0d0dc6f) will **decrease** coverage by `0.56%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3073 +/- ## - Coverage 55.14% 54.57% -0.57% - Complexity 3866 4194 +328 Files 488 542 +54 Lines 2361926750+3131 Branches 2528 3043 +515 + Hits 1302414600+1576 - Misses 943710708+1271 - Partials 1158 1442 +284 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (+0.40%)` | :arrow_up: | | hudiclient | `∅ <ø> (∅)` | | | hudicommon | `48.21% <ø> (-2.13%)` | :arrow_down: | | hudiflink | `59.94% <ø> (-3.31%)` | :arrow_down: | | hudihadoopmr | `51.34% <ø> (-0.09%)` | :arrow_down: | | hudisparkdatasource | `67.06% <ø> (-7.24%)` | :arrow_down: | | hudisync | `54.05% <ø> (+7.45%)` | :arrow_up: | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `72.37% <ø> (+1.48%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...g/apache/hudi/utilities/schema/SchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlci5qYXZh) | `71.42% <0.00%> (-28.58%)` | :arrow_down: | | [.../org/apache/hudi/sink/compact/CompactFunction.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdEZ1bmN0aW9uLmphdmE=) | `86.66% <0.00%> (-13.34%)` | :arrow_down: | | [...e/hudi/sink/partitioner/profile/WriteProfiles.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlcy5qYXZh) | `46.80% <0.00%> (-13.20%)` | :arrow_down: | | [...apache/hudi/sink/compact/CompactionCommitSink.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvbkNvbW1pdFNpbmsuamF2YQ==) | `66.66% <0.00%> (-8.34%)` | :arrow_down: | | [.../main/scala/org/apache/hudi/HoodieSparkUtils.scala](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVNwYXJrVXRpbHMuc2NhbGE=) | `84.26% <0.00%> (-7.31%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
codecov-commenter edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-860337193 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3074](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (c6c523b) into [master](https://codecov.io/gh/apache/hudi/commit/7d9f9d7d8241bfb70d50c557b0194cc8a87b6ee7?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (7d9f9d7) will **decrease** coverage by `38.96%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3074 +/- ## = - Coverage 55.04% 16.08% -38.97% + Complexity 4029 470 -3559 = Files 526 280 -246 Lines 2546610875-14591 Branches 2886 1106 -1780 = - Hits 14018 1749-12269 + Misses10057 8968 -1089 + Partials 1391 158 -1233 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (∅)` | | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-46.08%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `57.62% <ø> (-13.34%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [.../apache/hudi/hive/MultiPartKeysValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTXVsdGlQYXJ0S2V5c1ZhbHVlRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-90.91%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/SchemaDifference.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2NoZW1hRGlmZmVyZW5jZS5qYXZh) | `0.00% <0.00%> (-84.85%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3073: [HUDI-2006] Adding more yaml templates to test suite
codecov-commenter edited a comment on pull request #3073: URL: https://github.com/apache/hudi/pull/3073#issuecomment-860281795 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3073](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6d4ffe0) into [master](https://codecov.io/gh/apache/hudi/commit/0d0dc6fb07e0c5496224c75052ab4f43d57b40f6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0d0dc6f) will **decrease** coverage by `4.17%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3073 +/- ## - Coverage 55.14% 50.96% -4.18% + Complexity 3866 451-3415 Files 488 78 -410 Lines 23619 3255 -20364 Branches 2528 395-2133 - Hits 13024 1659 -11365 + Misses 9437 1444-7993 + Partials 1158 152-1006 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `∅ <ø> (∅)` | | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-41.22%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `72.37% <ø> (+1.48%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [.../apache/hudi/hive/MultiPartKeysValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTXVsdGlQYXJ0S2V5c1ZhbHVlRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-90.91%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/SchemaDifference.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2NoZW1hRGlmZmVyZW5jZS5qYXZh) | `0.00% <0.00%> (-84.85%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
codecov-commenter edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-864499795 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3117](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (72a3594) into [master](https://codecov.io/gh/apache/hudi/commit/c08fbb4268ee4b227452fd27d5e6ba322eeef00e?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (c08fbb4) will **decrease** coverage by `42.94%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3117 +/- ## - Coverage 46.01% 3.06% -42.95% + Complexity 5306 82 -5224 Files 911 280 -631 Lines 39476 10881-28595 Branches 42541107 -3147 - Hits 18166 334-17832 + Misses19456 10521 -8935 + Partials 1854 26 -1828 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-30.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-46.35%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.46% <ø> (-48.92%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-17.01%)` | :arrow_down: | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
hudi-bot edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-864499549 ## CI report: * 6d4b556a0c3cf88b7d097a9b036ad36eb0e7820b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=545) * 72a359459c780ae3d5f0e6b598aa363513ca28ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=548) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
hudi-bot edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-864499549 ## CI report: * 6d4b556a0c3cf88b7d097a9b036ad36eb0e7820b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=545) * 72a359459c780ae3d5f0e6b598aa363513ca28ca UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
codecov-commenter edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-860337193 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3074](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (c6c523b) into [master](https://codecov.io/gh/apache/hudi/commit/7d9f9d7d8241bfb70d50c557b0194cc8a87b6ee7?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (7d9f9d7) will **decrease** coverage by `38.96%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3074 +/- ## = - Coverage 55.04% 16.08% -38.97% + Complexity 4029 470 -3559 = Files 526 280 -246 Lines 2546610875-14591 Branches 2886 1106 -1780 = - Hits 14018 1749-12269 + Misses10057 8968 -1089 + Partials 1391 158 -1233 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (∅)` | | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-46.08%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `57.62% <ø> (-13.34%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3074?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [.../apache/hudi/hive/MultiPartKeysValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTXVsdGlQYXJ0S2V5c1ZhbHVlRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-90.91%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/SchemaDifference.java](https://codecov.io/gh/apache/hudi/pull/3074/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2NoZW1hRGlmZmVyZW5jZS5qYXZh) | `0.00% <0.00%> (-84.85%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3073: [HUDI-2006] Adding more yaml templates to test suite
codecov-commenter edited a comment on pull request #3073: URL: https://github.com/apache/hudi/pull/3073#issuecomment-860281795 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3073](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6d4ffe0) into [master](https://codecov.io/gh/apache/hudi/commit/0d0dc6fb07e0c5496224c75052ab4f43d57b40f6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (0d0dc6f) will **decrease** coverage by `4.17%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3073 +/- ## - Coverage 55.14% 50.96% -4.18% + Complexity 3866 451-3415 Files 488 78 -410 Lines 23619 3255 -20364 Branches 2528 395-2133 - Hits 13024 1659 -11365 + Misses 9437 1444-7993 + Partials 1158 152-1006 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `?` | | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-41.22%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `72.37% <ø> (+1.48%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3073?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [.../apache/hudi/hive/MultiPartKeysValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTXVsdGlQYXJ0S2V5c1ZhbHVlRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-90.91%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/SchemaDifference.java](https://codecov.io/gh/apache/hudi/pull/3073/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2NoZW1hRGlmZmVyZW5jZS5qYXZh) | `0.00% <0.00%> (-84.85%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
hudi-bot edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-861653947 ## CI report: * c6c523b32beca10c5797c9a3374ac1c387184744 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=547) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
codecov-commenter edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-864499795 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3117](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6d4b556) into [master](https://codecov.io/gh/apache/hudi/commit/c08fbb4268ee4b227452fd27d5e6ba322eeef00e?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (c08fbb4) will **increase** coverage by `20.83%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3117 +/- ## = + Coverage 46.01% 66.85% +20.83% + Complexity 5306 788 -4518 = Files 911 96 -815 Lines 39476 3451-36025 Branches 4254 352 -3902 = - Hits 18166 2307-15859 + Misses19456 1007-18449 + Partials 1854 137 -1717 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `66.85% <ø> (+36.40%)` | :arrow_up: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3117?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...metadata/SparkHoodieBackedTableMetadataWriter.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvU3BhcmtIb29kaWVCYWNrZWRUYWJsZU1ldGFkYXRhV3JpdGVyLmphdmE=) | `72.36% <0.00%> (-0.88%)` | :arrow_down: | | [...g/apache/hudi/table/action/BaseActionExecutor.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9CYXNlQWN0aW9uRXhlY3V0b3IuamF2YQ==) | | | | [...g/apache/hudi/common/config/LockConfiguration.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9Mb2NrQ29uZmlndXJhdGlvbi5qYXZh) | | | | [...di/index/bloom/ListBasedGlobalIndexFileFilter.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2luZGV4L2Jsb29tL0xpc3RCYXNlZEdsb2JhbEluZGV4RmlsZUZpbHRlci5qYXZh) | | | | [.../main/scala/org/apache/hudi/cli/SparkHelpers.scala](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL2NsaS9TcGFya0hlbHBlcnMuc2NhbGE=) | | | | [...pache/hudi/cli/commands/FileSystemViewCommand.java](https://codecov.io/gh/apache/hudi/pull/3117/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL0ZpbGVTeXN0ZW1WaWV3Q29tbWFuZC5qYXZh) | | | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3073: [HUDI-2006] Adding more yaml templates to test suite
hudi-bot edited a comment on pull request #3073: URL: https://github.com/apache/hudi/pull/3073#issuecomment-861653904 ## CI report: * 6d4ffe00b540244e50d742ec845897b95a9b64a7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=546) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
hudi-bot edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-861653947 ## CI report: * 37197b78a1a8ead38d37a760df920cb53ec451de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=360) * c6c523b32beca10c5797c9a3374ac1c387184744 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=547) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3074: [HUDI-2007] Fixing hudi_test_suite for spark nodes and adding spark bulk_insert node
hudi-bot edited a comment on pull request #3074: URL: https://github.com/apache/hudi/pull/3074#issuecomment-861653947 ## CI report: * 37197b78a1a8ead38d37a760df920cb53ec451de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=360) * c6c523b32beca10c5797c9a3374ac1c387184744 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3172: [HUDI-2069] Enable custom avro deserializer by default and refactored String constants
codecov-commenter edited a comment on pull request #3172: URL: https://github.com/apache/hudi/pull/3172#issuecomment-870958818 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3073: [HUDI-2006] Adding more yaml templates to test suite
hudi-bot edited a comment on pull request #3073: URL: https://github.com/apache/hudi/pull/3073#issuecomment-861653904 ## CI report: * e039f490bf755cc7b29a1446aa8156dfd3adc9c4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=189) * 6d4ffe00b540244e50d742ec845897b95a9b64a7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=546) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3073: [HUDI-2006] Adding more yaml templates to test suite
hudi-bot edited a comment on pull request #3073: URL: https://github.com/apache/hudi/pull/3073#issuecomment-861653904 ## CI report: * e039f490bf755cc7b29a1446aa8156dfd3adc9c4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=189) * 6d4ffe00b540244e50d742ec845897b95a9b64a7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3073: [HUDI-2006] Adding more yaml templates to test suite
nsivabalan commented on pull request #3073: URL: https://github.com/apache/hudi/pull/3073#issuecomment-870973213 I have tested all the yamls. I have updated more into to README. will merge once CI succeeds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3172: [HUDI-2069] Enable custom avro deserializer by default and refactored String constants
codecov-commenter edited a comment on pull request #3172: URL: https://github.com/apache/hudi/pull/3172#issuecomment-870958818 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3172?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3172](https://codecov.io/gh/apache/hudi/pull/3172?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3adabda) into [master](https://codecov.io/gh/apache/hudi/commit/b8a8f572d6a760c744b0a5228f2512d3d113d390?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b8a8f57) will **decrease** coverage by `25.82%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3172?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3172 +/- ## = - Coverage 44.25% 18.43% -25.83% + Complexity 4598 870 -3728 = Files 825 376 -449 Lines 3658014327-22253 Branches 3942 1459 -2483 = - Hits 16189 2641-13548 + Misses1864511523 -7122 + Partials 1746 163 -1583 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (+5.84%)` | :arrow_up: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.45% <0.00%> (-48.99%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3172?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...i/utilities/deser/KafkaAvroSchemaDeserializer.java](https://codecov.io/gh/apache/hudi/pull/3172/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2Rlc2VyL0thZmthQXZyb1NjaGVtYURlc2VyaWFsaXplci5qYXZh) | `0.00% <0.00%> (-81.25%)` | :arrow_down: | | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3172/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3172/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.10%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3172/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3172/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] codecov-commenter commented on pull request #3172: [HUDI-2069] Enable custom avro deserializer by default and refactored String constants
codecov-commenter commented on pull request #3172: URL: https://github.com/apache/hudi/pull/3172#issuecomment-870958818 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3172?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3172](https://codecov.io/gh/apache/hudi/pull/3172?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3adabda) into [master](https://codecov.io/gh/apache/hudi/commit/b8a8f572d6a760c744b0a5228f2512d3d113d390?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b8a8f57) will **decrease** coverage by `41.18%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3172?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3172 +/- ## - Coverage 44.25% 3.07% -41.19% + Complexity 4598 82 -4516 Files 825 280 -545 Lines 36580 10876-25704 Branches 39421107 -2835 - Hits 16189 334-15855 + Misses18645 10516 -8129 + Partials 1746 26 -1720 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-16.45%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.45% <0.00%> (-48.99%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3172?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...i/utilities/deser/KafkaAvroSchemaDeserializer.java](https://codecov.io/gh/apache/hudi/pull/3172/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2Rlc2VyL0thZmthQXZyb1NjaGVtYURlc2VyaWFsaXplci5qYXZh) | `0.00% <0.00%> (-81.25%)` | :arrow_down: | | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3172/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/3172/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.10%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3172/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3172/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3172: [HUDI-2069] Enable custom avro deserializer by default and refactored String constants
hudi-bot edited a comment on pull request #3172: URL: https://github.com/apache/hudi/pull/3172#issuecomment-869593797 ## CI report: * 3adabdab2a50423966fc3c94edf29ba7fefe31c1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=544) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch asf-site updated: Travis CI build asf-site
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 61ff78d Travis CI build asf-site 61ff78d is described below commit 61ff78dd928c7dda445d649ccb00f86c221ad3fb Author: CI AuthorDate: Tue Jun 29 21:30:39 2021 + Travis CI build asf-site --- content/cn/docs/configurations.html | 8 ++-- content/docs/configurations.html| 21 + 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/content/cn/docs/configurations.html b/content/cn/docs/configurations.html index 57429e1..4d459d5 100644 --- a/content/cn/docs/configurations.html +++ b/content/cn/docs/configurations.html @@ -452,8 +452,12 @@ 如果设置为true，则生成基于Hive格式的partition目录：=/span KEYGENERATOR_CLASS_OPT_KEY -属性：hoodie.datasource.write.keygenerator.class, 默认值：org.apache.hudi.keygen.SimpleKeyGenerator - 键生成器类，实现从输入的Row对象中提取键 +属性：hoodie.datasource.write.keygenerator.class + 键生成器类，实现从输入的Row对象中提取键。该配置优先级大于 hoodie.datasource.write.keygenerator.type, 用于使用用户自定义键生成器 + +KEYGENERATOR_TYPE_OPT_KEY +属性: hoodie.datasource.write.keygenerator.type, 默认值: SIMPLE +键生成器类型，默认 SIMPLE 类型，该配置优先级低于 hoodie.datasource.write.keygenerator.class, 是推荐使用的配置方式 COMMIT_METADATA_KEYPREFIX_OPT_KEY 属性：hoodie.datasource.write.commitmeta.key.prefix, 默认值：_ diff --git a/content/docs/configurations.html b/content/docs/configurations.html index 55b641f..e3440a8 100644 --- a/content/docs/configurations.html +++ b/content/docs/configurations.html @@ -499,8 +499,15 @@ The actual datasource level configs are listed below. KEYGENERATOR_CLASS_OPT_KEY hoodie.datasource.write.keygenerator.class NO -org.apache.hudi.keygen.SimpleKeyGenerator -Key generator class, that implements will extract the key out of incoming Row object. +No default +Key generator class, that implements will extract the key out of incoming Row object. This config has higher precedence over keygen type, and it is used for user-defined KeyGenerator . + + +KEYGENERATOR_TYPE_OPT_KEY +hoodie.datasource.write.keygenerator.type +NO +SIMPLE +Key generator type, , that indicate which KeyGenerator to use. This is the recommended configuration option for key generator, and has lower priority than KEYGENERATOR_CLASS_OPT_KEY . COMMIT_METADATA_KEYPREFIX_OPT_KEY @@ -739,8 +746,14 @@ The actual datasource level configs are listed below. hoodie.datasource.write.keygenerator.class N -SimpleAvroKeyGenerator.class - Key generator class, that implements will extract the key out of incoming record +No default + Key generator class, that implements will extract the key out of incoming Row object. This config has higher precedence over keygen type, and it is used for user-defined KeyGenerator + + +hoodie.datasource.write.keygenerator.type +N +SIMPLE + Key generator type, that indicate which KeyGenerator to use. This is the recommended configuration option for key generator, and has lower priority than KEYGENERATOR_CLASS_OPT_KEY write.partition.url_encode
[GitHub] [hudi] hudi-bot edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
hudi-bot edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-864499549 ## CI report: * 6d4b556a0c3cf88b7d097a9b036ad36eb0e7820b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=545) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
hudi-bot edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-864499549 ## CI report: * dd248a3ab08bd58980bf731e24d8e403b6f64c17 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=385) * 6d4b556a0c3cf88b7d097a9b036ad36eb0e7820b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] sbernauer commented on a change in pull request #3172: [HUDI-2069] Enable custom avro deserializer by default and refactored String constants
sbernauer commented on a change in pull request #3172: URL: https://github.com/apache/hudi/pull/3172#discussion_r660951463 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java ## @@ -60,7 +66,7 @@ public AvroKafkaSource(TypedProperties props, JavaSparkContext sparkContext, Spa super(props, sparkContext, sparkSession, schemaProvider); props.put(NATIVE_KAFKA_KEY_DESERIALIZER_PROP, StringDeserializer.class); -String deserializerClassName = props.getString(DataSourceWriteOptions.KAFKA_AVRO_VALUE_DESERIALIZER(), ""); +String deserializerClassName = props.getString(KAFKA_AVRO_VALUE_DESERIALIZER_OPT_KEY(), DEFAULT_KAFKA_AVRO_VALUE_DESERIALIZER_OPT_VAL()); Review comment: Hi, i would prefer to leave the block in - even if we turn the feature on by default. Doing so we have the possibility to turn the feature explicitly off (`hoodie.deltastreamer.source.kafka.value.deserializer.class=`) if it causes any problems -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3172: [HUDI-2069] Enable custom avro deserializer by default and refactored String constants
hudi-bot edited a comment on pull request #3172: URL: https://github.com/apache/hudi/pull/3172#issuecomment-869593797 ## CI report: * 72a2badac20bc0a42b1ca5975bfc342f0f43bc90 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=543) * 3adabdab2a50423966fc3c94edf29ba7fefe31c1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=544) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3172: [HUDI-2069] Enable custom avro deserializer by default and refactored String constants
hudi-bot edited a comment on pull request #3172: URL: https://github.com/apache/hudi/pull/3172#issuecomment-869593797 ## CI report: * 72a2badac20bc0a42b1ca5975bfc342f0f43bc90 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=543) * 3adabdab2a50423966fc3c94edf29ba7fefe31c1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] sbernauer commented on a change in pull request #3172: [HUDI-2069] Enable custom avro deserializer by default and refactored String constants
sbernauer commented on a change in pull request #3172: URL: https://github.com/apache/hudi/pull/3172#discussion_r660949292 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java ## @@ -52,6 +54,10 @@ // these are native kafka's config. do not change the config names. private static final String NATIVE_KAFKA_KEY_DESERIALIZER_PROP = "key.deserializer"; private static final String NATIVE_KAFKA_VALUE_DESERIALIZER_PROP = "value.deserializer"; + // These are settings used to pass things to KafkaAvroDeserializer + public static final String KAFKA_AVRO_VALUE_DESERIALIZER_SCHEMA = "hoodie.deltastreamer.source.kafka.value.deserializer.schema"; + public static final String KAFKA_AVRO_VALUE_DESERIALIZER_PROPERTY_PREFIX = "hoodie.deltastreamer.source.kafka.value.deserializer."; Review comment: Thanks for the suggestion, fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] sbernauer commented on a change in pull request #3172: [HUDI-2069] Enable custom avro deserializer by default and refactored String constants
sbernauer commented on a change in pull request #3172:
URL: https://github.com/apache/hudi/pull/3172#discussion_r660948842
##
File path:
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##
@@ -381,7 +381,6 @@ object DataSourceWriteOptions {
val DEFAULT_ASYNC_COMPACT_ENABLE_OPT_VAL = "true"
// Avro Kafka Source configs
- val KAFKA_AVRO_VALUE_DESERIALIZER =
"hoodie.deltastreamer.source.kafka.value.deserializer.class"
- // Schema to be used in custom kakfa deserializer
- val KAFKA_AVRO_VALUE_DESERIALIZER_SCHEMA =
"hoodie.deltastreamer.source.kafka.value.deserializer.schema"
+ val KAFKA_AVRO_VALUE_DESERIALIZER_OPT_KEY =
"hoodie.deltastreamer.source.kafka.value.deserializer.class"
+ val DEFAULT_KAFKA_AVRO_VALUE_DESERIALIZER_OPT_VAL =
"org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer"
Review comment:
Alrigth, seems perfectly fine to me. Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] rmahindra123 commented on a change in pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
rmahindra123 commented on a change in pull request #3117:
URL: https://github.com/apache/hudi/pull/3117#discussion_r660933216
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/collection/SpillableRocksDBBasedMap.java
##
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.collection;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieNotSupportedException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.jetbrains.annotations.NotNull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.AbstractMap;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * This class provides a disk spillable only map implementation that is based
on RocksDB.
+ */
+public final class SpillableRocksDBBasedMap implements SpillableDiskMap {
+ // ColumnFamily allows partitioning data within RockDB, which allows
+ // independent configuration and faster deletes across partitions
+ // https://github.com/facebook/rocksdb/wiki/Column-Families
+ // For this use case, we use a single static column family/ partition
+ //
+ private static final String COLUMN_FAMILY_NAME = "spill_map";
+
+ private static final Logger LOG =
LogManager.getLogger(SpillableRocksDBBasedMap.class);
+ // Stores the key and corresponding value's latest metadata spilled to disk
+ private final Set keySet;
+ private final String rocksDbStoragePath;
+ private RocksDBDAO rocksDb;
+
+ public SpillableRocksDBBasedMap(String rocksDbStoragePath) throws
IOException {
+this.keySet = new HashSet<>();
+this.rocksDbStoragePath = rocksDbStoragePath;
+ }
+
+ @Override
+ public int size() {
+return keySet.size();
+ }
+
+ @Override
+ public boolean isEmpty() {
+return keySet.isEmpty();
+ }
+
+ @Override
+ public boolean containsKey(Object key) {
+return keySet.contains((T) key);
+ }
+
+ @Override
+ public boolean containsValue(Object value) {
+throw new HoodieNotSupportedException("unable to compare values in map");
+ }
+
+ @Override
+ public R get(Object key) {
+if (!containsKey(key)) {
+ return null;
+}
+return getRocksDb().get(COLUMN_FAMILY_NAME, (T) key);
+ }
+
+ @Override
+ public R put(T key, R value) {
+getRocksDb().put(COLUMN_FAMILY_NAME, key, value);
+keySet.add(key);
+return value;
+ }
+
+ @Override
+ public R remove(Object key) {
+R value = get(key);
+if (value != null) {
+ keySet.remove((T) key);
+ getRocksDb().delete(COLUMN_FAMILY_NAME, (T) key);
+}
+return value;
+ }
+
+ @Override
+ public void putAll(Map keyValues) {
+getRocksDb().writeBatch(batch -> keyValues.forEach((key, value) ->
getRocksDb().putInBatch(batch, COLUMN_FAMILY_NAME, key, value)));
+keySet.addAll(keyValues.keySet());
+ }
+
+ @Override
+ public void clear() {
+close();
+ }
+
+ @Override
+ public @NotNull Set keySet() {
+return keySet;
+ }
+
+ @Override
+ public @NotNull Collection values() {
+throw new HoodieException("Unsupported Operation Exception");
+ }
+
+ @Override
+ public @NotNull Set> entrySet() {
+Set> entrySet = new HashSet<>();
+for (T key : keySet) {
+ entrySet.add(new AbstractMap.SimpleEntry<>(key, get(key)));
+}
+return entrySet;
+ }
+
+ /**
+ * Custom iterator to iterate over values written to disk.
+ */
+ @Override
+ public @NotNull Iterator iterator() {
+return getRocksDb().iterator(COLUMN_FAMILY_NAME);
+ }
+
+ @Override
+ public Stream valueStream() {
+return keySet.stream().sorted().sequential().map(valueMetaData -> (R)
get(valueMetaData));
Review comment:
Good catch. I made it consistent with DiskBasedMap, but in disk based
map we do it to sort the offsets for ensuring sequential reads. But for
rocksDb, we do not need to do it and leave it to RocksDb to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
[GitHub] [hudi] nsivabalan commented on pull request #3183: [HUDI-2090]when hudi metadata is enabled, use different users to quer…
nsivabalan commented on pull request #3183: URL: https://github.com/apache/hudi/pull/3183#issuecomment-870859594 @n3nash @bvaradar : how come we did not encounter this until now. any thoughts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3172: [HUDI-2069] Enable custom avro deserializer by default and refactored String constants
nsivabalan commented on a change in pull request #3172:
URL: https://github.com/apache/hudi/pull/3172#discussion_r660903610
##
File path:
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##
@@ -381,7 +381,6 @@ object DataSourceWriteOptions {
val DEFAULT_ASYNC_COMPACT_ENABLE_OPT_VAL = "true"
// Avro Kafka Source configs
- val KAFKA_AVRO_VALUE_DESERIALIZER =
"hoodie.deltastreamer.source.kafka.value.deserializer.class"
- // Schema to be used in custom kakfa deserializer
- val KAFKA_AVRO_VALUE_DESERIALIZER_SCHEMA =
"hoodie.deltastreamer.source.kafka.value.deserializer.schema"
+ val KAFKA_AVRO_VALUE_DESERIALIZER_OPT_KEY =
"hoodie.deltastreamer.source.kafka.value.deserializer.class"
+ val DEFAULT_KAFKA_AVRO_VALUE_DESERIALIZER_OPT_VAL =
"org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer"
Review comment:
I would prefer to make the new deserializer default in next release. by
that time, I assume atleast you would have tested it and would be running it in
prod and so we know it works smoothly end to end.
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java
##
@@ -60,7 +66,7 @@ public AvroKafkaSource(TypedProperties props,
JavaSparkContext sparkContext, Spa
super(props, sparkContext, sparkSession, schemaProvider);
props.put(NATIVE_KAFKA_KEY_DESERIALIZER_PROP, StringDeserializer.class);
-String deserializerClassName =
props.getString(DataSourceWriteOptions.KAFKA_AVRO_VALUE_DESERIALIZER(), "");
+String deserializerClassName =
props.getString(KAFKA_AVRO_VALUE_DESERIALIZER_OPT_KEY(),
DEFAULT_KAFKA_AVRO_VALUE_DESERIALIZER_OPT_VAL());
Review comment:
we need to be careful in line 71. since we have updated the default
value, we may never go into if condition of line 71. Can you fix that please.
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java
##
@@ -52,6 +54,10 @@
// these are native kafka's config. do not change the config names.
private static final String NATIVE_KAFKA_KEY_DESERIALIZER_PROP =
"key.deserializer";
private static final String NATIVE_KAFKA_VALUE_DESERIALIZER_PROP =
"value.deserializer";
+ // These are settings used to pass things to KafkaAvroDeserializer
+ public static final String KAFKA_AVRO_VALUE_DESERIALIZER_SCHEMA =
"hoodie.deltastreamer.source.kafka.value.deserializer.schema";
+ public static final String KAFKA_AVRO_VALUE_DESERIALIZER_PROPERTY_PREFIX =
"hoodie.deltastreamer.source.kafka.value.deserializer.";
Review comment:
can we move this ahead of line 58 and use this to form the other
variables.
for eg:
```
public static final String KAFKA_AVRO_VALUE_DESERIALIZER_SCHEMA =
KAFKA_AVRO_VALUE_DESERIALIZER_PROPERTY_PREFIX + "schema";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] rmahindra123 edited a comment on pull request #3117: [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap
rmahindra123 edited a comment on pull request #3117: URL: https://github.com/apache/hudi/pull/3117#issuecomment-870852282 > I see we have more test coverage in TestExternalSpillableMap. Can we parametrize the tests that are applicable and run those for both type of spillable maps. I had tests for TestExternalSpillableMap in the stacked diff, anyway added it here. > In general, do we have tests around diff values for maxInMemorySizeInBytes for external spillable map. If not, do you think we can add them while we are at this. We do test if the right amount of keys were in memory and the right amount was spilled over. Wondering how different values of maxInMemorySizeInBytes will help? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
