[jira] [Commented] (HUDI-2185) Remove the default parallelism of index bootstrap and bucket assigner
[ https://issues.apache.org/jira/browse/HUDI-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381826#comment-17381826 ] ASF GitHub Bot commented on HUDI-2185: -- codecov-commenter commented on pull request #3286: URL: https://github.com/apache/hudi/pull/3286#issuecomment-881213773 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3286](https://codecov.io/gh/apache/hudi/pull/3286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ffecf0d) into [master](https://codecov.io/gh/apache/hudi/commit/3b264e80d9ccc58ebd0dff3b00f50ad9d481e03c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b264e8) will **decrease** coverage by `44.98%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3286 +/- ## - Coverage 47.81% 2.83% -44.99% + Complexity 5568 85 -5483 Files 936 284 -652 Lines 41624 11828-29796 Branches 4188 981 -3207 - Hits 19903 335-19568 + Misses19957 11467 -8490 + Partials 1764 26 -1738 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.52%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-50.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term
[GitHub] [hudi] codecov-commenter commented on pull request #3286: [HUDI-2185] Remove the default parallelism of index bootstrap and buc…
codecov-commenter commented on pull request #3286: URL: https://github.com/apache/hudi/pull/3286#issuecomment-881213773 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3286](https://codecov.io/gh/apache/hudi/pull/3286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ffecf0d) into [master](https://codecov.io/gh/apache/hudi/commit/3b264e80d9ccc58ebd0dff3b00f50ad9d481e03c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b264e8) will **decrease** coverage by `44.98%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3286 +/- ## - Coverage 47.81% 2.83% -44.99% + Complexity 5568 85 -5483 Files 936 284 -652 Lines 41624 11828-29796 Branches 4188 981 -3207 - Hits 19903 335-19568 + Misses19957 11467 -8490 + Partials 1764 26 -1738 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.52%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-50.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381825#comment-17381825 ] ASF GitHub Bot commented on HUDI-1447: -- codecov-commenter edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-850284847 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2438](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e98b8e4) into [master](https://codecov.io/gh/apache/hudi/commit/5804ad8e32ae05758ebc5e47f5d4fb4db371ab52?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5804ad8) will **decrease** coverage by `31.70%`. > The diff coverage is `85.45%`. [](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2438 +/- ## = - Coverage 47.72% 16.02% -31.71% + Complexity 5528 502 -5026 = Files 934 284 -650 Lines 4145711869-29588 Branches 4166 986 -3180 = - Hits 19786 1902-17884 + Misses19914 9802-10112 + Partials 1757 165 -1592 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-49.64%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.70% <85.45%> (+0.44%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.72% <83.33%> (+1.15%)` | :arrow_up: | | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.00% <87.23%> (-0.68%)` | :arrow_down: | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `100.00% <100.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_c
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
codecov-commenter edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-850284847 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2438](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e98b8e4) into [master](https://codecov.io/gh/apache/hudi/commit/5804ad8e32ae05758ebc5e47f5d4fb4db371ab52?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5804ad8) will **decrease** coverage by `31.70%`. > The diff coverage is `85.45%`. [](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2438 +/- ## = - Coverage 47.72% 16.02% -31.71% + Complexity 5528 502 -5026 = Files 934 284 -650 Lines 4145711869-29588 Branches 4166 986 -3180 = - Hits 19786 1902-17884 + Misses19914 9802-10112 + Partials 1757 165 -1592 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-49.64%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.70% <85.45%> (+0.44%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.72% <83.33%> (+1.15%)` | :arrow_up: | | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `87.00% <87.23%> (-0.68%)` | :arrow_down: | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `100.00% <100.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/
[jira] [Commented] (HUDI-2185) Remove the default parallelism of index bootstrap and bucket assigner
[ https://issues.apache.org/jira/browse/HUDI-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381824#comment-17381824 ] ASF GitHub Bot commented on HUDI-2185: -- hudi-bot edited a comment on pull request #3286: URL: https://github.com/apache/hudi/pull/3286#issuecomment-881209345 ## CI report: * 91d7f902a24b23455ef6033640aa17285ade5a2b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=960) * ffecf0d2ae0461d67d9d4a18555b75f6d2f39a0e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove the default parallelism of index bootstrap and bucket assigner > - > > Key: HUDI-2185 > URL: https://issues.apache.org/jira/browse/HUDI-2185 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3286: [HUDI-2185] Remove the default parallelism of index bootstrap and buc…
hudi-bot edited a comment on pull request #3286: URL: https://github.com/apache/hudi/pull/3286#issuecomment-881209345 ## CI report: * 91d7f902a24b23455ef6033640aa17285ade5a2b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=960) * ffecf0d2ae0461d67d9d4a18555b75f6d2f39a0e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1138) Re-implement marker files via timeline server
[
https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381822#comment-17381822
]
ASF GitHub Bot commented on HUDI-1138:
--
yihua commented on a change in pull request #3233:
URL: https://github.com/apache/hudi/pull/3233#discussion_r670998935
##
File path:
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/MarkerHandler.java
##
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.timeline.service.handlers;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.model.IOType;
+import org.apache.hudi.common.table.view.FileSystemViewManager;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.javalin.Context;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.timeline.service.RequestHandler.jsonifyResult;
+
+public class MarkerHandler extends Handler {
+ public static final String MARKERS_FILENAME = "MARKERS";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final Logger LOG = LogManager.getLogger(MarkerHandler.class);
+ private static final long MARGIN_TIME_MS = 10L;
+
+ private final Registry metricsRegistry;
+ private final ScheduledExecutorService executorService;
+ // {markerDirPath -> all markers}
+ private final Map> allMarkersMap = new HashMap<>();
+ // {markerDirPath -> {markerFileIndex -> markers}}
+ private final Map> fileMarkersMap = new
HashMap<>();
+ private final List createMarkerFutures = new
ArrayList<>();
+ private final List isMarkerFileInUseList;
+ private final long batchIntervalMs;
+ private final int parallelism;
+ private volatile Object createMarkerRequestlockObject = new Object();
+ private long nextBatchProcessTimeMs = 0L;
+
+ public MarkerHandler(Configuration conf, FileSystem fileSystem,
FileSystemViewManager viewManager, Registry metricsRegistry,
+ int batchNumThreads, long batchIntervalMs, int
parallelism) throws IOException {
+super(conf, fileSystem, viewManager);
+LOG.info("*** MarkerHandler FileSystem: " + this.fileSystem.getScheme());
+LOG.info("*** MarkerHandler Params: batchNumThreads=" + batchNumThreads +
" batchIntervalMs=" + batchIntervalMs + "ms");
+this.metricsRegistry = metricsRegistry;
+this.batchIntervalMs = batchIntervalMs;
+this.parallelism = parallelism;
+this.executorService = Executors.newScheduledThreadPool(batchNumThreads);
+List isMarkerFileInUseList = new ArrayList<>(batchNumThreads);
+for (int i = 0; i < batchNumThreads; i++) {
+ isMarkerFileInUseList.add(false);
+}
+this.isMarkerFileInUseList =
Collections.synchronizedList(isMarkerFileInUseList);
+ }
+
+ public Set getAllMarkers(String markerDirPath) {
+return allMarkersMap.getOrDefault(markerDirPath, new HashSet<>());
+ }
+
+ public Set getCreateAndMergeMarkers(String markerDirPath) {
+return allMarkersMap.getOrDefault(markerDirPath, new HashSet<>()).stre
[jira] [Commented] (HUDI-2185) Remove the default parallelism of index bootstrap and bucket assigner
[ https://issues.apache.org/jira/browse/HUDI-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381821#comment-17381821 ] ASF GitHub Bot commented on HUDI-2185: -- hudi-bot edited a comment on pull request #3286: URL: https://github.com/apache/hudi/pull/3286#issuecomment-881209345 ## CI report: * 91d7f902a24b23455ef6033640aa17285ade5a2b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=960) * ffecf0d2ae0461d67d9d4a18555b75f6d2f39a0e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove the default parallelism of index bootstrap and bucket assigner > - > > Key: HUDI-2185 > URL: https://issues.apache.org/jira/browse/HUDI-2185 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] yihua commented on a change in pull request #3233: [WIP][HUDI-1138] Add timeline-server-based marker file strategy for improving marker-related latency
yihua commented on a change in pull request #3233:
URL: https://github.com/apache/hudi/pull/3233#discussion_r670998935
##
File path:
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/MarkerHandler.java
##
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.timeline.service.handlers;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.model.IOType;
+import org.apache.hudi.common.table.view.FileSystemViewManager;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.javalin.Context;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.timeline.service.RequestHandler.jsonifyResult;
+
+public class MarkerHandler extends Handler {
+ public static final String MARKERS_FILENAME = "MARKERS";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final Logger LOG = LogManager.getLogger(MarkerHandler.class);
+ private static final long MARGIN_TIME_MS = 10L;
+
+ private final Registry metricsRegistry;
+ private final ScheduledExecutorService executorService;
+ // {markerDirPath -> all markers}
+ private final Map> allMarkersMap = new HashMap<>();
+ // {markerDirPath -> {markerFileIndex -> markers}}
+ private final Map> fileMarkersMap = new
HashMap<>();
+ private final List createMarkerFutures = new
ArrayList<>();
+ private final List isMarkerFileInUseList;
+ private final long batchIntervalMs;
+ private final int parallelism;
+ private volatile Object createMarkerRequestlockObject = new Object();
+ private long nextBatchProcessTimeMs = 0L;
+
+ public MarkerHandler(Configuration conf, FileSystem fileSystem,
FileSystemViewManager viewManager, Registry metricsRegistry,
+ int batchNumThreads, long batchIntervalMs, int
parallelism) throws IOException {
+super(conf, fileSystem, viewManager);
+LOG.info("*** MarkerHandler FileSystem: " + this.fileSystem.getScheme());
+LOG.info("*** MarkerHandler Params: batchNumThreads=" + batchNumThreads +
" batchIntervalMs=" + batchIntervalMs + "ms");
+this.metricsRegistry = metricsRegistry;
+this.batchIntervalMs = batchIntervalMs;
+this.parallelism = parallelism;
+this.executorService = Executors.newScheduledThreadPool(batchNumThreads);
+List isMarkerFileInUseList = new ArrayList<>(batchNumThreads);
+for (int i = 0; i < batchNumThreads; i++) {
+ isMarkerFileInUseList.add(false);
+}
+this.isMarkerFileInUseList =
Collections.synchronizedList(isMarkerFileInUseList);
+ }
+
+ public Set getAllMarkers(String markerDirPath) {
+return allMarkersMap.getOrDefault(markerDirPath, new HashSet<>());
+ }
+
+ public Set getCreateAndMergeMarkers(String markerDirPath) {
+return allMarkersMap.getOrDefault(markerDirPath, new HashSet<>()).stream()
+.filter(markerName -> !markerName.endsWith(IOType.APPEND.name()))
+.collect(Collectors.toSet());
+ }
+
+ public CompletableFuture createMarker(Context context, String
markerDirPath, String markerName) {
+LOG.info("Request:
[GitHub] [hudi] hudi-bot edited a comment on pull request #3286: [HUDI-2185] Remove the default parallelism of index bootstrap and buc…
hudi-bot edited a comment on pull request #3286: URL: https://github.com/apache/hudi/pull/3286#issuecomment-881209345 ## CI report: * 91d7f902a24b23455ef6033640aa17285ade5a2b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=960) * ffecf0d2ae0461d67d9d4a18555b75f6d2f39a0e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381820#comment-17381820 ] ASF GitHub Bot commented on HUDI-1447: -- liujinhui1994 commented on a change in pull request #2438: URL: https://github.com/apache/hudi/pull/2438#discussion_r670998409 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java ## @@ -282,6 +301,36 @@ private Long delayOffsetCalculation(Option lastCheckpointStr, Set DeltaStreamer kafka source supports consuming from specified timestamp > -- > > Key: HUDI-1447 > URL: https://issues.apache.org/jira/browse/HUDI-1447 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer >Reporter: wangxianghu#1 >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:high, user-support-issues > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
liujinhui1994 commented on a change in pull request #2438: URL: https://github.com/apache/hudi/pull/2438#discussion_r670998409 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java ## @@ -282,6 +301,36 @@ private Long delayOffsetCalculation(Option lastCheckpointStr, Set
[GitHub] [hudi] hudi-bot edited a comment on pull request #3286: [HUDI-2185] Remove the default parallelism of index bootstrap and buc…
hudi-bot edited a comment on pull request #3286: URL: https://github.com/apache/hudi/pull/3286#issuecomment-881209345 ## CI report: * 91d7f902a24b23455ef6033640aa17285ade5a2b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=960) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2185) Remove the default parallelism of index bootstrap and bucket assigner
[ https://issues.apache.org/jira/browse/HUDI-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381819#comment-17381819 ] ASF GitHub Bot commented on HUDI-2185: -- hudi-bot edited a comment on pull request #3286: URL: https://github.com/apache/hudi/pull/3286#issuecomment-881209345 ## CI report: * 91d7f902a24b23455ef6033640aa17285ade5a2b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=960) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove the default parallelism of index bootstrap and bucket assigner > - > > Key: HUDI-2185 > URL: https://issues.apache.org/jira/browse/HUDI-2185 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[
https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381818#comment-17381818
]
ASF GitHub Bot commented on HUDI-1447:
--
liujinhui1994 commented on a change in pull request #2438:
URL: https://github.com/apache/hudi/pull/2438#discussion_r670997475
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestKafkaSource.java
##
@@ -193,7 +193,7 @@ public void testJsonKafkaSourceWithDefaultUpperCap() {
Source jsonSource = new JsonKafkaSource(props, jsc, sparkSession,
schemaProvider, metrics);
SourceFormatAdapter kafkaSource = new SourceFormatAdapter(jsonSource);
-Config.maxEventsFromKafkaSource = 500;
+//props.setProperty("hoodie.deltastreamer.kafka.source.maxEvents", "500");
Review comment:
//props.setProperty("hoodie.deltastreamer.kafka.source.maxEvents",
"500")
should not appear here.
sorry,My Mistake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DeltaStreamer kafka source supports consuming from specified timestamp
> --
>
> Key: HUDI-1447
> URL: https://issues.apache.org/jira/browse/HUDI-1447
> Project: Apache Hudi
> Issue Type: New Feature
> Components: DeltaStreamer
>Reporter: wangxianghu#1
>Assignee: liujinhui
>Priority: Major
> Labels: pull-request-available, sev:high, user-support-issues
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
liujinhui1994 commented on a change in pull request #2438:
URL: https://github.com/apache/hudi/pull/2438#discussion_r670997475
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestKafkaSource.java
##
@@ -193,7 +193,7 @@ public void testJsonKafkaSourceWithDefaultUpperCap() {
Source jsonSource = new JsonKafkaSource(props, jsc, sparkSession,
schemaProvider, metrics);
SourceFormatAdapter kafkaSource = new SourceFormatAdapter(jsonSource);
-Config.maxEventsFromKafkaSource = 500;
+//props.setProperty("hoodie.deltastreamer.kafka.source.maxEvents", "500");
Review comment:
//props.setProperty("hoodie.deltastreamer.kafka.source.maxEvents",
"500")
should not appear here.
sorry,My Mistake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[
https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381816#comment-17381816
]
ASF GitHub Bot commented on HUDI-1447:
--
liujinhui1994 commented on a change in pull request #2438:
URL: https://github.com/apache/hudi/pull/2438#discussion_r670996945
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##
@@ -212,6 +234,9 @@ public KafkaOffsetGen(TypedProperties props) {
Set topicPartitions = partitionInfoList.stream()
.map(x -> new TopicPartition(x.topic(),
x.partition())).collect(Collectors.toSet());
+ if (Config.KAFKA_CHECKPOINT_TYPE_TIMESTAMP.equals(kafkaCheckpointType)
&& isValidCheckpointType(lastCheckpointStr)) {
+lastCheckpointStr = getOffsetsByTimestamp(consumer, partitionInfoList,
topicPartitions, topicName, Long.parseLong(lastCheckpointStr.get()));
+ }
Review comment:
I don't think there needs to be an "else if" here.
If you are using timestamp kafkaCheckpointType, lastCheckpointStr will be
passed a timestamp of "timestamp type", which we will handle using the
getOffsetByTimestamp method.
If it is not a timestamp type, then we can interpret it as a regular string
type checkpoint rule and do not process it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DeltaStreamer kafka source supports consuming from specified timestamp
> --
>
> Key: HUDI-1447
> URL: https://issues.apache.org/jira/browse/HUDI-1447
> Project: Apache Hudi
> Issue Type: New Feature
> Components: DeltaStreamer
>Reporter: wangxianghu#1
>Assignee: liujinhui
>Priority: Major
> Labels: pull-request-available, sev:high, user-support-issues
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] liujinhui1994 commented on a change in pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
liujinhui1994 commented on a change in pull request #2438:
URL: https://github.com/apache/hudi/pull/2438#discussion_r670996945
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##
@@ -212,6 +234,9 @@ public KafkaOffsetGen(TypedProperties props) {
Set topicPartitions = partitionInfoList.stream()
.map(x -> new TopicPartition(x.topic(),
x.partition())).collect(Collectors.toSet());
+ if (Config.KAFKA_CHECKPOINT_TYPE_TIMESTAMP.equals(kafkaCheckpointType)
&& isValidCheckpointType(lastCheckpointStr)) {
+lastCheckpointStr = getOffsetsByTimestamp(consumer, partitionInfoList,
topicPartitions, topicName, Long.parseLong(lastCheckpointStr.get()));
+ }
Review comment:
I don't think there needs to be an "else if" here.
If you are using timestamp kafkaCheckpointType, lastCheckpointStr will be
passed a timestamp of "timestamp type", which we will handle using the
getOffsetByTimestamp method.
If it is not a timestamp type, then we can interpret it as a regular string
type checkpoint rule and do not process it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2185) Remove the default parallelism of index bootstrap and bucket assigner
[ https://issues.apache.org/jira/browse/HUDI-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381815#comment-17381815 ] ASF GitHub Bot commented on HUDI-2185: -- hudi-bot commented on pull request #3286: URL: https://github.com/apache/hudi/pull/3286#issuecomment-881209345 ## CI report: * 91d7f902a24b23455ef6033640aa17285ade5a2b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove the default parallelism of index bootstrap and bucket assigner > - > > Key: HUDI-2185 > URL: https://issues.apache.org/jira/browse/HUDI-2185 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot commented on pull request #3286: [HUDI-2185] Remove the default parallelism of index bootstrap and buc…
hudi-bot commented on pull request #3286: URL: https://github.com/apache/hudi/pull/3286#issuecomment-881209345 ## CI report: * 91d7f902a24b23455ef6033640aa17285ade5a2b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2185) Remove the default parallelism of index bootstrap and bucket assigner
[ https://issues.apache.org/jira/browse/HUDI-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381814#comment-17381814 ] ASF GitHub Bot commented on HUDI-2185: -- danny0405 opened a new pull request #3286: URL: https://github.com/apache/hudi/pull/3286 …ket assigner ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove the default parallelism of index bootstrap and bucket assigner > - > > Key: HUDI-2185 > URL: https://issues.apache.org/jira/browse/HUDI-2185 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2185) Remove the default parallelism of index bootstrap and bucket assigner
[ https://issues.apache.org/jira/browse/HUDI-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2185: - Labels: pull-request-available (was: ) > Remove the default parallelism of index bootstrap and bucket assigner > - > > Key: HUDI-2185 > URL: https://issues.apache.org/jira/browse/HUDI-2185 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] danny0405 opened a new pull request #3286: [HUDI-2185] Remove the default parallelism of index bootstrap and buc…
danny0405 opened a new pull request #3286: URL: https://github.com/apache/hudi/pull/3286 …ket assigner ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-2185) Remove the default parallelism of index bootstrap and bucket assigner
Danny Chen created HUDI-2185: Summary: Remove the default parallelism of index bootstrap and bucket assigner Key: HUDI-2185 URL: https://issues.apache.org/jira/browse/HUDI-2185 Project: Apache Hudi Issue Type: Improvement Components: Flink Integration Reporter: Danny Chen Assignee: Danny Chen Fix For: 0.9.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2169) Remove keyby when write.operation is Insert
[ https://issues.apache.org/jira/browse/HUDI-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381813#comment-17381813 ] ASF GitHub Bot commented on HUDI-2169: -- danny0405 commented on pull request #3265: URL: https://github.com/apache/hudi/pull/3265#issuecomment-881207185 Please wait for my confirm before merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove keyby when write.operation is Insert > --- > > Key: HUDI-2169 > URL: https://issues.apache.org/jira/browse/HUDI-2169 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: yuzhaojing >Assignee: yuzhaojing >Priority: Minor > Labels: pull-request-available > > When write.operation is Insert, user can tolerate data duplication or data > that does not need to be merged. In this case, the keyby is unnecessary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] danny0405 commented on pull request #3265: [HUDI-2169] Remove keyby when write.operation is Insert
danny0405 commented on pull request #3265: URL: https://github.com/apache/hudi/pull/3265#issuecomment-881207185 Please wait for my confirm before merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381809#comment-17381809
]
ASF GitHub Bot commented on HUDI-1860:
--
nsivabalan commented on a change in pull request #3184:
URL: https://github.com/apache/hudi/pull/3184#discussion_r670990427
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##
@@ -455,6 +459,12 @@ public void refreshTimeline() throws IOException {
case BULK_INSERT:
writeStatusRDD = writeClient.bulkInsert(records, instantTime);
break;
+ case INSERT_OVERWRITE:
+writeStatusRDD = writeClient.insertOverwrite(records,
instantTime).getWriteStatuses();
+break;
+ case INSERT_OVERWRITE_TABLE:
Review comment:
Do we plan to add "Delete_partition" in a follow up PR? Not looking to
expand the scope of this PR. just asking
##
File path:
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/HoodieTestSuiteWriter.java
##
@@ -163,6 +163,26 @@ private boolean allowWriteClientAccess(DagNode dagNode) {
}
}
+ public JavaRDD insertOverwrite(Option instantTime)
throws Exception {
Review comment:
awesome in fixing test suite as well :)
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -1695,6 +1695,54 @@ public void
testJdbcSourceIncrementalFetchInContinuousMode() {
}
}
+ @Test
+ public void testInsertOverwrite() throws Exception {
+String tableBasePath = dfsBasePath + "/insert_overwrite";
+// Initial insert
+HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath,
WriteOperationType.INSERT);
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// No new data => no commits.
+cfg.sourceLimit = 0;
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// insert overwrite
+cfg.sourceLimit = 1000;
+cfg.operation = WriteOperationType.INSERT_OVERWRITE;
Review comment:
- I see lot of commonality between both these tests. Can we reuse code
as much as possible.
- InsertOverwrite overwrites only the partitions matching the incoming
records. But guess we are not really testing that. Do you think we can test
this scenario.
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##
@@ -475,8 +485,8 @@ public void refreshTimeline() throws IOException {
LOG.warn("Some records failed to be merged but forcing commit since
commitOnErrors set. Errors/Total="
+ totalErrorRecords + "/" + totalRecords);
}
-
- boolean success = writeClient.commit(instantTime, writeStatusRDD,
Option.of(checkpointCommitMetadata));
+ String commitActionType = CommitUtils.getCommitActionType(cfg.operation,
HoodieTableType.valueOf(cfg.tableType));
Review comment:
Since you folks did some investigation around this, Do you think we need
to improve docs on
WriteClient.startCommitWithTime(). There are two overloaded methods. may be
clearly calling out when to call which one would benefit all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add INSERT_OVERWRITE support to DeltaStreamer
> -
>
> Key: HUDI-1860
> URL: https://issues.apache.org/jira/browse/HUDI-1860
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: Sagar Sumit
>Assignee: Samrat Deb
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> As discussed in [this
> RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller],
> having full fetch mode use the inser_overwrite to write to sync would be
> better as it can handle schema changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
nsivabalan commented on a change in pull request #3184:
URL: https://github.com/apache/hudi/pull/3184#discussion_r670990427
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##
@@ -455,6 +459,12 @@ public void refreshTimeline() throws IOException {
case BULK_INSERT:
writeStatusRDD = writeClient.bulkInsert(records, instantTime);
break;
+ case INSERT_OVERWRITE:
+writeStatusRDD = writeClient.insertOverwrite(records,
instantTime).getWriteStatuses();
+break;
+ case INSERT_OVERWRITE_TABLE:
Review comment:
Do we plan to add "Delete_partition" in a follow up PR? Not looking to
expand the scope of this PR. just asking
##
File path:
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/HoodieTestSuiteWriter.java
##
@@ -163,6 +163,26 @@ private boolean allowWriteClientAccess(DagNode dagNode) {
}
}
+ public JavaRDD insertOverwrite(Option instantTime)
throws Exception {
Review comment:
awesome in fixing test suite as well :)
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -1695,6 +1695,54 @@ public void
testJdbcSourceIncrementalFetchInContinuousMode() {
}
}
+ @Test
+ public void testInsertOverwrite() throws Exception {
+String tableBasePath = dfsBasePath + "/insert_overwrite";
+// Initial insert
+HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath,
WriteOperationType.INSERT);
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// No new data => no commits.
+cfg.sourceLimit = 0;
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// insert overwrite
+cfg.sourceLimit = 1000;
+cfg.operation = WriteOperationType.INSERT_OVERWRITE;
Review comment:
- I see lot of commonality between both these tests. Can we reuse code
as much as possible.
- InsertOverwrite overwrites only the partitions matching the incoming
records. But guess we are not really testing that. Do you think we can test
this scenario.
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##
@@ -475,8 +485,8 @@ public void refreshTimeline() throws IOException {
LOG.warn("Some records failed to be merged but forcing commit since
commitOnErrors set. Errors/Total="
+ totalErrorRecords + "/" + totalRecords);
}
-
- boolean success = writeClient.commit(instantTime, writeStatusRDD,
Option.of(checkpointCommitMetadata));
+ String commitActionType = CommitUtils.getCommitActionType(cfg.operation,
HoodieTableType.valueOf(cfg.tableType));
Review comment:
Since you folks did some investigation around this, Do you think we need
to improve docs on
WriteClient.startCommitWithTime(). There are two overloaded methods. may be
clearly calling out when to call which one would benefit all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381807#comment-17381807 ] ASF GitHub Bot commented on HUDI-1447: -- codecov-commenter edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-850284847 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2438](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e98b8e4) into [master](https://codecov.io/gh/apache/hudi/commit/5804ad8e32ae05758ebc5e47f5d4fb4db371ab52?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5804ad8) will **decrease** coverage by `44.90%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2438 +/- ## - Coverage 47.72% 2.82% -44.91% + Complexity 5528 85 -5443 Files 934 284 -650 Lines 41457 11869-29588 Branches 4166 986 -3180 - Hits 19786 335-19451 + Misses19914 11508 -8406 + Partials 1757 26 -1731 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-49.64%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.99% <0.00%> (-50.27%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-71.57%)` | :arrow_down: | | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.69%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&ut
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
codecov-commenter edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-850284847 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2438](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e98b8e4) into [master](https://codecov.io/gh/apache/hudi/commit/5804ad8e32ae05758ebc5e47f5d4fb4db371ab52?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5804ad8) will **decrease** coverage by `44.90%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2438 +/- ## - Coverage 47.72% 2.82% -44.91% + Complexity 5528 85 -5443 Files 934 284 -650 Lines 41457 11869-29588 Branches 4166 986 -3180 - Hits 19786 335-19451 + Misses19914 11508 -8406 + Partials 1757 26 -1731 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-49.64%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.99% <0.00%> (-50.27%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-71.57%)` | :arrow_down: | | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `0.00% <0.00%> (-87.69%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/ut
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381803#comment-17381803 ] ASF GitHub Bot commented on HUDI-1447: -- liujinhui1994 commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-881203019 > Let's try to land this in by weekend. Its been hanging for quite sometime. ok. Sorry, I'll deal with it now, please excuse me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DeltaStreamer kafka source supports consuming from specified timestamp > -- > > Key: HUDI-1447 > URL: https://issues.apache.org/jira/browse/HUDI-1447 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer >Reporter: wangxianghu#1 >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:high, user-support-issues > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] liujinhui1994 commented on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
liujinhui1994 commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-881203019 > Let's try to land this in by weekend. Its been hanging for quite sometime. ok. Sorry, I'll deal with it now, please excuse me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381801#comment-17381801 ] ASF GitHub Bot commented on HUDI-1860: -- nsivabalan commented on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-881202427 Can you fix "How did you verify the Pull Request" section in the desc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Samrat Deb >Priority: Major > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381799#comment-17381799 ] ASF GitHub Bot commented on HUDI-1771: -- codecov-commenter edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881152765 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3285](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a517fd1) into [master](https://codecov.io/gh/apache/hudi/commit/a62a6cff32e7d7f62ffdf6362fb8876497fd0528?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a62a6cf) will **decrease** coverage by `21.01%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3285 +/- ## = - Coverage 47.70% 26.69% -21.02% + Complexity 5550 1295 -4255 = Files 936 386 -550 Lines 4162415732-25892 Branches 4188 1355 -2833 = - Hits 19858 4199-15659 + Misses211229 -8771 + Partials 1766 304 -1462 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `20.27% <50.00%> (-14.19%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUNyZWF0ZUhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...rg/apache/hudi/client/model/HoodieInternalRow.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L21vZGVsL0hvb2RpZUludGVybmFsUm93LmphdmE=) | `79.48% <87.50%> (+1.70%)` | :arrow_up: | | [...java/org/apache/hudi/io/HoodieRowCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments
[GitHub] [hudi] nsivabalan commented on pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
nsivabalan commented on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-881202427 Can you fix "How did you verify the Pull Request" section in the desc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
codecov-commenter edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881152765 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3285](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a517fd1) into [master](https://codecov.io/gh/apache/hudi/commit/a62a6cff32e7d7f62ffdf6362fb8876497fd0528?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a62a6cf) will **decrease** coverage by `21.01%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3285 +/- ## = - Coverage 47.70% 26.69% -21.02% + Complexity 5550 1295 -4255 = Files 936 386 -550 Lines 4162415732-25892 Branches 4188 1355 -2833 = - Hits 19858 4199-15659 + Misses211229 -8771 + Partials 1766 304 -1462 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `20.27% <50.00%> (-14.19%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUNyZWF0ZUhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...rg/apache/hudi/client/model/HoodieInternalRow.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L21vZGVsL0hvb2RpZUludGVybmFsUm93LmphdmE=) | `79.48% <87.50%> (+1.70%)` | :arrow_up: | | [...java/org/apache/hudi/io/HoodieRowCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllUm93Q3JlYXRlSGFuZGxlLmphdmE=) | `92.77% <100.00%> (+0.27%)` | :arrow_up: | | [...main/java/org/apache/hudi/me
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381798#comment-17381798 ] ASF GitHub Bot commented on HUDI-1447: -- hudi-bot edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-863310563 ## CI report: * b77b63994db2e91853a06d3a5c4c129a21feefcf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=863) * e98b8e407f1bbcd0f0219d2f2d65f4e95f663c00 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=959) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DeltaStreamer kafka source supports consuming from specified timestamp > -- > > Key: HUDI-1447 > URL: https://issues.apache.org/jira/browse/HUDI-1447 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer >Reporter: wangxianghu#1 >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:high, user-support-issues > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
hudi-bot edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-863310563 ## CI report: * b77b63994db2e91853a06d3a5c4c129a21feefcf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=863) * e98b8e407f1bbcd0f0219d2f2d65f4e95f663c00 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=959) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
nsivabalan commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-881200974 Let's try to land this in by weekend. Its been hanging for quite sometime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381797#comment-17381797 ] ASF GitHub Bot commented on HUDI-1447: -- nsivabalan commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-881200974 Let's try to land this in by weekend. Its been hanging for quite sometime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DeltaStreamer kafka source supports consuming from specified timestamp > -- > > Key: HUDI-1447 > URL: https://issues.apache.org/jira/browse/HUDI-1447 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer >Reporter: wangxianghu#1 >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:high, user-support-issues > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381795#comment-17381795 ] ASF GitHub Bot commented on HUDI-1447: -- hudi-bot edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-863310563 ## CI report: * b77b63994db2e91853a06d3a5c4c129a21feefcf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=863) * e98b8e407f1bbcd0f0219d2f2d65f4e95f663c00 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DeltaStreamer kafka source supports consuming from specified timestamp > -- > > Key: HUDI-1447 > URL: https://issues.apache.org/jira/browse/HUDI-1447 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer >Reporter: wangxianghu#1 >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:high, user-support-issues > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[
https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381796#comment-17381796
]
ASF GitHub Bot commented on HUDI-1447:
--
nsivabalan commented on a change in pull request #2438:
URL: https://github.com/apache/hudi/pull/2438#discussion_r670978737
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##
@@ -212,6 +234,9 @@ public KafkaOffsetGen(TypedProperties props) {
Set topicPartitions = partitionInfoList.stream()
.map(x -> new TopicPartition(x.topic(),
x.partition())).collect(Collectors.toSet());
+ if (Config.KAFKA_CHECKPOINT_TYPE_TIMESTAMP.equals(kafkaCheckpointType)
&& isValidCheckpointType(lastCheckpointStr)) {
+lastCheckpointStr = getOffsetsByTimestamp(consumer, partitionInfoList,
topicPartitions, topicName, Long.parseLong(lastCheckpointStr.get()));
+ }
Review comment:
I was expecting a else if block after this line. Can you clarify please.
If not, we might go into the else block ?
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestKafkaOffsetGen.java
##
@@ -64,7 +63,7 @@ public void teardown() throws Exception {
private TypedProperties getConsumerConfigs(String autoOffsetReset) {
TypedProperties props = new TypedProperties();
-props.put(Config.KAFKA_AUTO_OFFSET_RESET, autoOffsetReset);
+props.put("auto.offset.reset", autoOffsetReset);
Review comment:
Do you think we can add some tests to this class for the timestamp type?
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestKafkaSource.java
##
@@ -193,7 +193,7 @@ public void testJsonKafkaSourceWithDefaultUpperCap() {
Source jsonSource = new JsonKafkaSource(props, jsc, sparkSession,
schemaProvider, metrics);
SourceFormatAdapter kafkaSource = new SourceFormatAdapter(jsonSource);
-Config.maxEventsFromKafkaSource = 500;
+//props.setProperty("hoodie.deltastreamer.kafka.source.maxEvents", "500");
Review comment:
why commented out?
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestKafkaSource.java
##
@@ -193,7 +193,7 @@ public void testJsonKafkaSourceWithDefaultUpperCap() {
Source jsonSource = new JsonKafkaSource(props, jsc, sparkSession,
schemaProvider, metrics);
SourceFormatAdapter kafkaSource = new SourceFormatAdapter(jsonSource);
-Config.maxEventsFromKafkaSource = 500;
+//props.setProperty("hoodie.deltastreamer.kafka.source.maxEvents", "500");
Review comment:
I tried your patch locally. the test fails if I uncomment this line. I
don't understand ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DeltaStreamer kafka source supports consuming from specified timestamp
> --
>
> Key: HUDI-1447
> URL: https://issues.apache.org/jira/browse/HUDI-1447
> Project: Apache Hudi
> Issue Type: New Feature
> Components: DeltaStreamer
>Reporter: wangxianghu#1
>Assignee: liujinhui
>Priority: Major
> Labels: pull-request-available, sev:high, user-support-issues
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
nsivabalan commented on a change in pull request #2438:
URL: https://github.com/apache/hudi/pull/2438#discussion_r670978737
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##
@@ -212,6 +234,9 @@ public KafkaOffsetGen(TypedProperties props) {
Set topicPartitions = partitionInfoList.stream()
.map(x -> new TopicPartition(x.topic(),
x.partition())).collect(Collectors.toSet());
+ if (Config.KAFKA_CHECKPOINT_TYPE_TIMESTAMP.equals(kafkaCheckpointType)
&& isValidCheckpointType(lastCheckpointStr)) {
+lastCheckpointStr = getOffsetsByTimestamp(consumer, partitionInfoList,
topicPartitions, topicName, Long.parseLong(lastCheckpointStr.get()));
+ }
Review comment:
I was expecting a else if block after this line. Can you clarify please.
If not, we might go into the else block ?
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestKafkaOffsetGen.java
##
@@ -64,7 +63,7 @@ public void teardown() throws Exception {
private TypedProperties getConsumerConfigs(String autoOffsetReset) {
TypedProperties props = new TypedProperties();
-props.put(Config.KAFKA_AUTO_OFFSET_RESET, autoOffsetReset);
+props.put("auto.offset.reset", autoOffsetReset);
Review comment:
Do you think we can add some tests to this class for the timestamp type?
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestKafkaSource.java
##
@@ -193,7 +193,7 @@ public void testJsonKafkaSourceWithDefaultUpperCap() {
Source jsonSource = new JsonKafkaSource(props, jsc, sparkSession,
schemaProvider, metrics);
SourceFormatAdapter kafkaSource = new SourceFormatAdapter(jsonSource);
-Config.maxEventsFromKafkaSource = 500;
+//props.setProperty("hoodie.deltastreamer.kafka.source.maxEvents", "500");
Review comment:
why commented out?
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestKafkaSource.java
##
@@ -193,7 +193,7 @@ public void testJsonKafkaSourceWithDefaultUpperCap() {
Source jsonSource = new JsonKafkaSource(props, jsc, sparkSession,
schemaProvider, metrics);
SourceFormatAdapter kafkaSource = new SourceFormatAdapter(jsonSource);
-Config.maxEventsFromKafkaSource = 500;
+//props.setProperty("hoodie.deltastreamer.kafka.source.maxEvents", "500");
Review comment:
I tried your patch locally. the test fails if I uncomment this line. I
don't understand ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
hudi-bot edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-863310563 ## CI report: * b77b63994db2e91853a06d3a5c4c129a21feefcf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=863) * e98b8e407f1bbcd0f0219d2f2d65f4e95f663c00 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381786#comment-17381786 ] ASF GitHub Bot commented on HUDI-1771: -- codecov-commenter edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881152765 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3285](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a517fd1) into [master](https://codecov.io/gh/apache/hudi/commit/a62a6cff32e7d7f62ffdf6362fb8876497fd0528?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a62a6cf) will **decrease** coverage by `21.02%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3285 +/- ## = - Coverage 47.70% 26.68% -21.03% + Complexity 5550 1294 -4256 = Files 936 386 -550 Lines 4162415732-25892 Branches 4188 1355 -2833 = - Hits 19858 4198-15660 + Misses211229 -8771 + Partials 1766 305 -1461 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `20.27% <50.00%> (-14.19%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.23% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUNyZWF0ZUhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...rg/apache/hudi/client/model/HoodieInternalRow.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L21vZGVsL0hvb2RpZUludGVybmFsUm93LmphdmE=) | `79.48% <87.50%> (+1.70%)` | :arrow_up: | | [...java/org/apache/hudi/io/HoodieRowCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
codecov-commenter edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881152765 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3285](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a517fd1) into [master](https://codecov.io/gh/apache/hudi/commit/a62a6cff32e7d7f62ffdf6362fb8876497fd0528?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a62a6cf) will **decrease** coverage by `21.02%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3285 +/- ## = - Coverage 47.70% 26.68% -21.03% + Complexity 5550 1294 -4256 = Files 936 386 -550 Lines 4162415732-25892 Branches 4188 1355 -2833 = - Hits 19858 4198-15660 + Misses211229 -8771 + Partials 1766 305 -1461 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `20.27% <50.00%> (-14.19%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.23% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUNyZWF0ZUhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...rg/apache/hudi/client/model/HoodieInternalRow.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L21vZGVsL0hvb2RpZUludGVybmFsUm93LmphdmE=) | `79.48% <87.50%> (+1.70%)` | :arrow_up: | | [...java/org/apache/hudi/io/HoodieRowCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllUm93Q3JlYXRlSGFuZGxlLmphdmE=) | `92.77% <100.00%> (+0.27%)` | :arrow_up: | | [...main/java/
[jira] [Commented] (HUDI-1884) MergeInto Support Partial Update For COW
[
https://issues.apache.org/jira/browse/HUDI-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381784#comment-17381784
]
ASF GitHub Bot commented on HUDI-1884:
--
hudi-bot edited a comment on pull request #3154:
URL: https://github.com/apache/hudi/pull/3154#issuecomment-868360998
## CI report:
* 63cf96134656dbb41eaab4e03e747776599255b6 UNKNOWN
* 82b4b8bef8d143a14d445ae35fa5ec8fbbd5889f UNKNOWN
* f5f0fd3f1eb1529643b7129349614ed80d106bed Azure:
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=956)
Bot commands
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MergeInto Support Partial Update For COW
> -
>
> Key: HUDI-1884
> URL: https://issues.apache.org/jira/browse/HUDI-1884
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
>
> Support partial update for merge-into statement, just like this:
> {code:java}
> merge into h0
> using s0
> on s0.id = h0.id
> when matched then update set price = s0.price + 10
> when not matched then insert *{code}
> Allow to update partial fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
hudi-bot edited a comment on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868360998 ## CI report: * 63cf96134656dbb41eaab4e03e747776599255b6 UNKNOWN * 82b4b8bef8d143a14d445ae35fa5ec8fbbd5889f UNKNOWN * f5f0fd3f1eb1529643b7129349614ed80d106bed Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=956) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1884) MergeInto Support Partial Update For COW
[
https://issues.apache.org/jira/browse/HUDI-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381783#comment-17381783
]
ASF GitHub Bot commented on HUDI-1884:
--
codecov-commenter edited a comment on pull request #3154:
URL: https://github.com/apache/hudi/pull/3154#issuecomment-868445474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MergeInto Support Partial Update For COW
> -
>
> Key: HUDI-1884
> URL: https://issues.apache.org/jira/browse/HUDI-1884
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
>
> Support partial update for merge-into statement, just like this:
> {code:java}
> merge into h0
> using s0
> on s0.id = h0.id
> when matched then update set price = s0.price + 10
> when not matched then insert *{code}
> Allow to update partial fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
codecov-commenter edited a comment on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868445474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2170) Always choose the latest record for HoodieRecordPayload
[ https://issues.apache.org/jira/browse/HUDI-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381782#comment-17381782 ] ASF GitHub Bot commented on HUDI-2170: -- codecov-commenter edited a comment on pull request #3267: URL: https://github.com/apache/hudi/pull/3267#issuecomment-878977860 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3267](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a275c3a) into [master](https://codecov.io/gh/apache/hudi/commit/01ad449ad68d9c77ee1493fcdf833df53df6106a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01ad449) will **decrease** coverage by `34.84%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3267 +/- ## = - Coverage 52.19% 17.35% -34.85% + Complexity 2659 884 -1775 = Files 335 386 +51 Lines 1498115336 +355 Branches 1505 1338 -167 = - Hits 7819 2661 -5158 - Misses 653612508 +5972 + Partials626 167 -459 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `20.91% <0.00%> (∅)` | | | hudicommon | `?` | | | hudihadoopmr | `?` | | | hudisync | `4.88% <ø> (?)` | | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-60.54%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/async/AsyncClusteringService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/async/HoodieAsyncService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ient/http/HoodieWriteCommitHttpCallbackClient.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NhbGxiYWNrL2NsaWVudC9odHRwL0hvb2RpZVdyaXRlQ29tbWl0SHR0cENhbGxiYWNrQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...udi/callback/util/HoodieCommitCallbackFactory.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3267: [HUDI-2170] Always choose the latest record for HoodieRecordPayload
codecov-commenter edited a comment on pull request #3267: URL: https://github.com/apache/hudi/pull/3267#issuecomment-878977860 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3267](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a275c3a) into [master](https://codecov.io/gh/apache/hudi/commit/01ad449ad68d9c77ee1493fcdf833df53df6106a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01ad449) will **decrease** coverage by `34.84%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3267 +/- ## = - Coverage 52.19% 17.35% -34.85% + Complexity 2659 884 -1775 = Files 335 386 +51 Lines 1498115336 +355 Branches 1505 1338 -167 = - Hits 7819 2661 -5158 - Misses 653612508 +5972 + Partials626 167 -459 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `20.91% <0.00%> (∅)` | | | hudicommon | `?` | | | hudihadoopmr | `?` | | | hudisync | `4.88% <ø> (?)` | | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-60.54%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/async/AsyncClusteringService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/async/HoodieAsyncService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ient/http/HoodieWriteCommitHttpCallbackClient.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NhbGxiYWNrL2NsaWVudC9odHRwL0hvb2RpZVdyaXRlQ29tbWl0SHR0cENhbGxiYWNrQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...udi/callback/util/HoodieCommitCallbackFactory.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NhbGxiYWNrL3V0aWwvSG9vZGllQ29tbWl0Q2FsbGJhY2tGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/client/AbstractClusteringClient.java](https://cod
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381780#comment-17381780 ] ASF GitHub Bot commented on HUDI-1771: -- swuferhong closed pull request #3285: URL: https://github.com/apache/hudi/pull/3285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381781#comment-17381781 ] ASF GitHub Bot commented on HUDI-1771: -- swuferhong opened a new pull request #3285: URL: https://github.com/apache/hudi/pull/3285 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request Propagate CDC format for hoodie. ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] swuferhong closed pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
swuferhong closed pull request #3285: URL: https://github.com/apache/hudi/pull/3285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381778#comment-17381778 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 2cf38df103e1435c4239824f4908520727e38f69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=958) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 2cf38df103e1435c4239824f4908520727e38f69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=958) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2170) Always choose the latest record for HoodieRecordPayload
[
https://issues.apache.org/jira/browse/HUDI-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381776#comment-17381776
]
ASF GitHub Bot commented on HUDI-2170:
--
hudi-bot edited a comment on pull request #3267:
URL: https://github.com/apache/hudi/pull/3267#issuecomment-878894177
## CI report:
* a275c3af0519430c6ff0dc5902dd84dc5d1a0882 Azure:
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=955)
Bot commands
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Always choose the latest record for HoodieRecordPayload
> ---
>
> Key: HUDI-2170
> URL: https://issues.apache.org/jira/browse/HUDI-2170
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Common Core
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Now in {{OverwriteWithLatestAvroPayload.preCombine}}, we still choose the old
> record when the new record has the same preCombine field with the old one,
> actually it is more natural to keep the new incoming record instead. The
> {{DefaultHoodieRecordPayload.combineAndGetUpdateValue}} method already did
> that.
> See issue: https://github.com/apache/hudi/issues/3266.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381777#comment-17381777 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 258dd4a9659b1edf6417ab645a3e93221cfcd2d6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=943) * 2cf38df103e1435c4239824f4908520727e38f69 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=958) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 258dd4a9659b1edf6417ab645a3e93221cfcd2d6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=943) * 2cf38df103e1435c4239824f4908520727e38f69 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=958) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3267: [HUDI-2170] Always choose the latest record for HoodieRecordPayload
hudi-bot edited a comment on pull request #3267: URL: https://github.com/apache/hudi/pull/3267#issuecomment-878894177 ## CI report: * a275c3af0519430c6ff0dc5902dd84dc5d1a0882 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=955) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381775#comment-17381775 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 258dd4a9659b1edf6417ab645a3e93221cfcd2d6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=943) * 2cf38df103e1435c4239824f4908520727e38f69 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 258dd4a9659b1edf6417ab645a3e93221cfcd2d6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=943) * 2cf38df103e1435c4239824f4908520727e38f69 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381773#comment-17381773 ] ASF GitHub Bot commented on HUDI-1771: -- swuferhong closed pull request #3285: URL: https://github.com/apache/hudi/pull/3285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381774#comment-17381774 ] ASF GitHub Bot commented on HUDI-1771: -- swuferhong opened a new pull request #3285: URL: https://github.com/apache/hudi/pull/3285 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request Propagate CDC format for hoodie. ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] swuferhong closed pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
swuferhong closed pull request #3285: URL: https://github.com/apache/hudi/pull/3285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[
https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381770#comment-17381770
]
ASF GitHub Bot commented on HUDI-2176:
--
nsivabalan commented on a change in pull request #3282:
URL: https://github.com/apache/hudi/pull/3282#discussion_r670964858
##
File path: hudi-cli/src/main/scala/org/apache/hudi/cli/SparkHelpers.scala
##
@@ -51,7 +51,8 @@ object SparkHelpers {
// Add current classLoad for config, if not will throw classNotFound of
'HoodieWrapperFileSystem'.
parquetConfig.getHadoopConf().setClassLoader(Thread.currentThread.getContextClassLoader)
-val writer = new HoodieParquetWriter[HoodieJsonPayload,
IndexedRecord](instantTime, destinationFile, parquetConfig, schema, new
SparkTaskContextSupplier())
+val writer = new HoodieParquetWriter[HoodieJsonPayload,
IndexedRecord](instantTime, destinationFile, parquetConfig, schema, new
SparkTaskContextSupplier(),
+ true)
Review comment:
yet to fix cli for virtual keys
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Virutal keys support for COW all operations
> ---
>
> Key: HUDI-2176
> URL: https://issues.apache.org/jira/browse/HUDI-2176
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
>Reporter: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Virutal keys support for COW all operations
> (merge handle)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
nsivabalan commented on a change in pull request #3282:
URL: https://github.com/apache/hudi/pull/3282#discussion_r670964858
##
File path: hudi-cli/src/main/scala/org/apache/hudi/cli/SparkHelpers.scala
##
@@ -51,7 +51,8 @@ object SparkHelpers {
// Add current classLoad for config, if not will throw classNotFound of
'HoodieWrapperFileSystem'.
parquetConfig.getHadoopConf().setClassLoader(Thread.currentThread.getContextClassLoader)
-val writer = new HoodieParquetWriter[HoodieJsonPayload,
IndexedRecord](instantTime, destinationFile, parquetConfig, schema, new
SparkTaskContextSupplier())
+val writer = new HoodieParquetWriter[HoodieJsonPayload,
IndexedRecord](instantTime, destinationFile, parquetConfig, schema, new
SparkTaskContextSupplier(),
+ true)
Review comment:
yet to fix cli for virtual keys
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[
https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381769#comment-17381769
]
ASF GitHub Bot commented on HUDI-2176:
--
nsivabalan commented on a change in pull request #3282:
URL: https://github.com/apache/hudi/pull/3282#discussion_r670966147
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java
##
@@ -278,7 +287,7 @@ protected boolean writeRecord(HoodieRecord hoodieRecord,
Option> latestBaseFiles) {
List>,
List, List>> hoodieKeyLocationFetchHandles =
-context.map(latestBaseFiles, partitionPathBaseFile -> new
HoodieKeyLocationFetchHandle<>(config, hoodieTable, partitionPathBaseFile),
parallelism);
+context.map(latestBaseFiles, partitionPathBaseFile -> new
HoodieKeyLocationFetchHandle<>(config, hoodieTable, partitionPathBaseFile,
null), parallelism);
Review comment:
as I have mentioned in the desc, I am not fixing flink and java in this
patch.
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestUpdateSchemaEvolution.java
##
@@ -121,7 +121,7 @@ private void
assertSchemaEvolutionOnUpdateResult(WriteStatus insertResult, Hoodi
jsc.parallelize(Arrays.asList(1)).map(x -> {
Executable executable = () -> {
HoodieMergeHandle mergeHandle = new
HoodieMergeHandle(updateTable.getConfig(), "101", updateTable,
-updateRecords.iterator(), updateRecords.get(0).getPartitionPath(),
insertResult.getFileId(), supplier);
+updateRecords.iterator(), updateRecords.get(0).getPartitionPath(),
insertResult.getFileId(), supplier, null);
Review comment:
Have not fixed this test for virtual keys
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java
##
@@ -142,6 +143,43 @@
return hoodieKeys;
}
+ /**
+ * Fetch {@link HoodieKey}s from the given parquet file.
+ *
+ * @param filePath The parquet file path.
+ * @param configuration configuration to build fs object
+ * @return {@link List} of {@link HoodieKey}s fetched from the parquet file
+ */
+ @Override
+ public List fetchRecordKeyPartitionPath(Configuration
configuration, Path filePath, BaseKeyGenerator keyGenerator) {
Review comment:
not sure if we can add another argument to existing api and
generate/fetch recordKeys and partition path based on that. Felt this is neat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Virutal keys support for COW all operations
> ---
>
> Key: HUDI-2176
> URL: https://issues.apache.org/jira/browse/HUDI-2176
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
>Reporter: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Virutal keys support for COW all operations
> (merge handle)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
nsivabalan commented on a change in pull request #3282:
URL: https://github.com/apache/hudi/pull/3282#discussion_r670966147
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java
##
@@ -278,7 +287,7 @@ protected boolean writeRecord(HoodieRecord hoodieRecord,
Option> latestBaseFiles) {
List>,
List, List>> hoodieKeyLocationFetchHandles =
-context.map(latestBaseFiles, partitionPathBaseFile -> new
HoodieKeyLocationFetchHandle<>(config, hoodieTable, partitionPathBaseFile),
parallelism);
+context.map(latestBaseFiles, partitionPathBaseFile -> new
HoodieKeyLocationFetchHandle<>(config, hoodieTable, partitionPathBaseFile,
null), parallelism);
Review comment:
as I have mentioned in the desc, I am not fixing flink and java in this
patch.
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestUpdateSchemaEvolution.java
##
@@ -121,7 +121,7 @@ private void
assertSchemaEvolutionOnUpdateResult(WriteStatus insertResult, Hoodi
jsc.parallelize(Arrays.asList(1)).map(x -> {
Executable executable = () -> {
HoodieMergeHandle mergeHandle = new
HoodieMergeHandle(updateTable.getConfig(), "101", updateTable,
-updateRecords.iterator(), updateRecords.get(0).getPartitionPath(),
insertResult.getFileId(), supplier);
+updateRecords.iterator(), updateRecords.get(0).getPartitionPath(),
insertResult.getFileId(), supplier, null);
Review comment:
Have not fixed this test for virtual keys
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java
##
@@ -142,6 +143,43 @@
return hoodieKeys;
}
+ /**
+ * Fetch {@link HoodieKey}s from the given parquet file.
+ *
+ * @param filePath The parquet file path.
+ * @param configuration configuration to build fs object
+ * @return {@link List} of {@link HoodieKey}s fetched from the parquet file
+ */
+ @Override
+ public List fetchRecordKeyPartitionPath(Configuration
configuration, Path filePath, BaseKeyGenerator keyGenerator) {
Review comment:
not sure if we can add another argument to existing api and
generate/fetch recordKeys and partition path based on that. Felt this is neat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2296: [HUDI-1425] Performance loss with the additional hoodieRecords.isEmpty() in HoodieSparkSqlWriter#write
hudi-bot edited a comment on pull request #2296: URL: https://github.com/apache/hudi/pull/2296#issuecomment-874367360 ## CI report: * d4e3d89e0c02f1b9a704a61627f7aafec14e67e6 UNKNOWN * dc36883644cf88971ecf13dc3642d60e60a1e6ec UNKNOWN * cd6767b606cfa19b064e4455f7a396e84e13d839 UNKNOWN * 7ebc9ff5395d8790339586ae3ff7e56b05a6a569 UNKNOWN * 4c86b7708e4204d585180ae8e9187c416ba6d052 UNKNOWN * fa8ba5e890fd394820a0ff91e0395b9be62d7ba8 UNKNOWN * 6ac9d507dc541a7e17741c3e1fcd8a6bb28aecc6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=954) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1425) Performance loss with the additional hoodieRecords.isEmpty() in HoodieSparkSqlWriter#write
[ https://issues.apache.org/jira/browse/HUDI-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381764#comment-17381764 ] ASF GitHub Bot commented on HUDI-1425: -- hudi-bot edited a comment on pull request #2296: URL: https://github.com/apache/hudi/pull/2296#issuecomment-874367360 ## CI report: * d4e3d89e0c02f1b9a704a61627f7aafec14e67e6 UNKNOWN * dc36883644cf88971ecf13dc3642d60e60a1e6ec UNKNOWN * cd6767b606cfa19b064e4455f7a396e84e13d839 UNKNOWN * 7ebc9ff5395d8790339586ae3ff7e56b05a6a569 UNKNOWN * 4c86b7708e4204d585180ae8e9187c416ba6d052 UNKNOWN * fa8ba5e890fd394820a0ff91e0395b9be62d7ba8 UNKNOWN * 6ac9d507dc541a7e17741c3e1fcd8a6bb28aecc6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=954) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Performance loss with the additional hoodieRecords.isEmpty() in > HoodieSparkSqlWriter#write > -- > > Key: HUDI-1425 > URL: https://issues.apache.org/jira/browse/HUDI-1425 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Affects Versions: 0.9.0 >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Blocker > Labels: pull-request-available > Fix For: 0.9.0 > > Attachments: 截屏2020-11-30 下午9.47.55.png > > > Currently in HoodieSparkSqlWriter#write, there is a _isEmpty()_ test for > _hoodieRecords._ This may be a heavy operator in the case when the > _hoodieRecords_ contains complex RDD operate. > !截屏2020-11-30 下午9.47.55.png|width=1255,height=161! > IMO this test does nothing to do with the performance improve，but rather > affects performance. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[
https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381761#comment-17381761
]
ASF GitHub Bot commented on HUDI-2176:
--
nsivabalan commented on a change in pull request #3282:
URL: https://github.com/apache/hudi/pull/3282#discussion_r670964858
##
File path: hudi-cli/src/main/scala/org/apache/hudi/cli/SparkHelpers.scala
##
@@ -51,7 +51,8 @@ object SparkHelpers {
// Add current classLoad for config, if not will throw classNotFound of
'HoodieWrapperFileSystem'.
parquetConfig.getHadoopConf().setClassLoader(Thread.currentThread.getContextClassLoader)
-val writer = new HoodieParquetWriter[HoodieJsonPayload,
IndexedRecord](instantTime, destinationFile, parquetConfig, schema, new
SparkTaskContextSupplier())
+val writer = new HoodieParquetWriter[HoodieJsonPayload,
IndexedRecord](instantTime, destinationFile, parquetConfig, schema, new
SparkTaskContextSupplier(),
+ true)
Review comment:
yet to fix cli's for virtual keys
##
File path: hudi-common/src/main/java/org/apache/hudi/common/util/OrcUtils.java
##
@@ -105,6 +107,11 @@
return hoodieKeys;
}
+ @Override
+ public List fetchRecordKeyPartitionPath(Configuration
configuration, Path filePath, BaseKeyGenerator keyGenerator) {
+return null;
Review comment:
Yet to add virtual keys support to Orc base files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Virutal keys support for COW all operations
> ---
>
> Key: HUDI-2176
> URL: https://issues.apache.org/jira/browse/HUDI-2176
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
>Reporter: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Virutal keys support for COW all operations
> (merge handle)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
nsivabalan commented on a change in pull request #3282:
URL: https://github.com/apache/hudi/pull/3282#discussion_r670964858
##
File path: hudi-cli/src/main/scala/org/apache/hudi/cli/SparkHelpers.scala
##
@@ -51,7 +51,8 @@ object SparkHelpers {
// Add current classLoad for config, if not will throw classNotFound of
'HoodieWrapperFileSystem'.
parquetConfig.getHadoopConf().setClassLoader(Thread.currentThread.getContextClassLoader)
-val writer = new HoodieParquetWriter[HoodieJsonPayload,
IndexedRecord](instantTime, destinationFile, parquetConfig, schema, new
SparkTaskContextSupplier())
+val writer = new HoodieParquetWriter[HoodieJsonPayload,
IndexedRecord](instantTime, destinationFile, parquetConfig, schema, new
SparkTaskContextSupplier(),
+ true)
Review comment:
yet to fix cli's for virtual keys
##
File path: hudi-common/src/main/java/org/apache/hudi/common/util/OrcUtils.java
##
@@ -105,6 +107,11 @@
return hoodieKeys;
}
+ @Override
+ public List fetchRecordKeyPartitionPath(Configuration
configuration, Path filePath, BaseKeyGenerator keyGenerator) {
+return null;
Review comment:
Yet to add virtual keys support to Orc base files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2170) Always choose the latest record for HoodieRecordPayload
[ https://issues.apache.org/jira/browse/HUDI-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381758#comment-17381758 ] ASF GitHub Bot commented on HUDI-2170: -- codecov-commenter edited a comment on pull request #3267: URL: https://github.com/apache/hudi/pull/3267#issuecomment-878977860 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3267](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a275c3a) into [master](https://codecov.io/gh/apache/hudi/commit/01ad449ad68d9c77ee1493fcdf833df53df6106a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01ad449) will **decrease** coverage by `49.36%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3267 +/- ## - Coverage 52.19% 2.83% -49.37% + Complexity 2659 85 -2574 Files 335 284 -51 Lines 14981 11828 -3153 Branches 1505 981 -524 - Hits 7819 335 -7484 - Misses 6536 11467 +4931 + Partials626 26 -600 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (∅)` | | | hudicommon | `?` | | | hudihadoopmr | `?` | | | hudisync | `4.88% <ø> (?)` | | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-60.54%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/async/AsyncClusteringService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/async/HoodieAsyncService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ient/http/HoodieWriteCommitHttpCallbackClient.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NhbGxiYWNrL2NsaWVudC9odHRwL0hvb2RpZVdyaXRlQ29tbWl0SHR0cENhbGxiYWNrQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...udi/callback/util/HoodieCommitCallbackFactory.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1j
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3267: [HUDI-2170] Always choose the latest record for HoodieRecordPayload
codecov-commenter edited a comment on pull request #3267: URL: https://github.com/apache/hudi/pull/3267#issuecomment-878977860 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3267](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a275c3a) into [master](https://codecov.io/gh/apache/hudi/commit/01ad449ad68d9c77ee1493fcdf833df53df6106a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01ad449) will **decrease** coverage by `49.36%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3267 +/- ## - Coverage 52.19% 2.83% -49.37% + Complexity 2659 85 -2574 Files 335 284 -51 Lines 14981 11828 -3153 Branches 1505 981 -524 - Hits 7819 335 -7484 - Misses 6536 11467 +4931 + Partials626 26 -600 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (∅)` | | | hudicommon | `?` | | | hudihadoopmr | `?` | | | hudisync | `4.88% <ø> (?)` | | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-60.54%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3267?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../org/apache/hudi/async/AsyncClusteringService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/async/HoodieAsyncService.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ient/http/HoodieWriteCommitHttpCallbackClient.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NhbGxiYWNrL2NsaWVudC9odHRwL0hvb2RpZVdyaXRlQ29tbWl0SHR0cENhbGxiYWNrQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...udi/callback/util/HoodieCommitCallbackFactory.java](https://codecov.io/gh/apache/hudi/pull/3267/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NhbGxiYWNrL3V0aWwvSG9vZGllQ29tbWl0Q2FsbGJhY2tGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/client/AbstractClusteringClient.java](https://codecov.io/gh/apa
[jira] [Commented] (HUDI-1884) MergeInto Support Partial Update For COW
[
https://issues.apache.org/jira/browse/HUDI-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381756#comment-17381756
]
ASF GitHub Bot commented on HUDI-1884:
--
hudi-bot edited a comment on pull request #3154:
URL: https://github.com/apache/hudi/pull/3154#issuecomment-868360998
## CI report:
* 63cf96134656dbb41eaab4e03e747776599255b6 UNKNOWN
* 82b4b8bef8d143a14d445ae35fa5ec8fbbd5889f UNKNOWN
* f5511851a56f5a23263bfb05690217718a329254 Azure:
[SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=940)
* f5f0fd3f1eb1529643b7129349614ed80d106bed Azure:
[PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=956)
Bot commands
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MergeInto Support Partial Update For COW
> -
>
> Key: HUDI-1884
> URL: https://issues.apache.org/jira/browse/HUDI-1884
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
>
> Support partial update for merge-into statement, just like this:
> {code:java}
> merge into h0
> using s0
> on s0.id = h0.id
> when matched then update set price = s0.price + 10
> when not matched then insert *{code}
> Allow to update partial fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
hudi-bot edited a comment on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868360998 ## CI report: * 63cf96134656dbb41eaab4e03e747776599255b6 UNKNOWN * 82b4b8bef8d143a14d445ae35fa5ec8fbbd5889f UNKNOWN * f5511851a56f5a23263bfb05690217718a329254 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=940) * f5f0fd3f1eb1529643b7129349614ed80d106bed Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=956) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1884) MergeInto Support Partial Update For COW
[ https://issues.apache.org/jira/browse/HUDI-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381753#comment-17381753 ] ASF GitHub Bot commented on HUDI-1884: -- codecov-commenter edited a comment on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868445474 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3154](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f5f0fd3) into [master](https://codecov.io/gh/apache/hudi/commit/a62a6cff32e7d7f62ffdf6362fb8876497fd0528?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a62a6cf) will **decrease** coverage by `44.87%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3154 +/- ## - Coverage 47.70% 2.83% -44.88% + Complexity 5550 85 -5465 Files 936 284 -652 Lines 41624 11835-29789 Branches 4188 981 -3207 - Hits 19858 335-19523 + Misses2 11474 -8526 + Partials 1766 26 -1740 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.47%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-50.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&u
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
codecov-commenter edited a comment on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868445474 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3154](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f5f0fd3) into [master](https://codecov.io/gh/apache/hudi/commit/a62a6cff32e7d7f62ffdf6362fb8876497fd0528?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a62a6cf) will **decrease** coverage by `44.87%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3154 +/- ## - Coverage 47.70% 2.83% -44.88% + Complexity 5550 85 -5465 Files 936 284 -652 Lines 41624 11835-29789 Branches 4188 981 -3207 - Hits 19858 335-19523 + Misses2 11474 -8526 + Partials 1766 26 -1740 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.47%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-50.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/utilities/sources/CsvDFSSource
[jira] [Resolved] (HUDI-1633) Make callback return HoodieWriteStat
[ https://issues.apache.org/jira/browse/HUDI-1633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianghu Wang resolved HUDI-1633. Resolution: Resolved Resolved via master branch : 3b264e80d9ccc58ebd0dff3b00f50ad9d481e03c > Make callback return HoodieWriteStat > > > Key: HUDI-1633 > URL: https://issues.apache.org/jira/browse/HUDI-1633 > Project: Apache Hudi > Issue Type: Improvement >Reporter: liujinhui >Assignee: liujinhui >Priority: Minor > Labels: pull-request-available, sev:normal > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1633) Make callback return HoodieWriteStat
[ https://issues.apache.org/jira/browse/HUDI-1633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381751#comment-17381751 ] ASF GitHub Bot commented on HUDI-1633: -- wangxianghu merged pull request #2445: URL: https://github.com/apache/hudi/pull/2445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make callback return HoodieWriteStat > > > Key: HUDI-1633 > URL: https://issues.apache.org/jira/browse/HUDI-1633 > Project: Apache Hudi > Issue Type: Improvement >Reporter: liujinhui >Assignee: liujinhui >Priority: Minor > Labels: pull-request-available, sev:normal > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[hudi] branch master updated (38cd74b -> 3b264e8)
This is an automated email from the ASF dual-hosted git repository. wangxianghu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 38cd74b [MINOR] Allow users to choose ORC as base file format in Spark SQL (#3279) add 3b264e8 [HUDI-1633] Make callback return HoodieWriteStat (#2445) No new revisions were added by this update. Summary of changes: .../hudi/callback/HoodieWriteCommitCallback.java | 5 .../http/HoodieWriteCommitHttpCallbackClient.java | 2 +- .../common/HoodieWriteCommitCallbackMessage.java | 33 +++--- .../impl/HoodieWriteCommitHttpCallback.java| 2 -- .../hudi/client/AbstractHoodieWriteClient.java | 2 +- .../kafka/HoodieWriteCommitKafkaCallback.java | 2 +- 6 files changed, 25 insertions(+), 21 deletions(-)
[GitHub] [hudi] wangxianghu merged pull request #2445: [HUDI-1633] Make callback return HoodieWriteStat
wangxianghu merged pull request #2445: URL: https://github.com/apache/hudi/pull/2445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2170) Always choose the latest record for HoodieRecordPayload
[
https://issues.apache.org/jira/browse/HUDI-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381750#comment-17381750
]
ASF GitHub Bot commented on HUDI-2170:
--
hudi-bot edited a comment on pull request #3267:
URL: https://github.com/apache/hudi/pull/3267#issuecomment-878894177
## CI report:
* 9c2e6e7b0efc3e235a9c736a095b4e0e003f8b1c Azure:
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=938)
* a275c3af0519430c6ff0dc5902dd84dc5d1a0882 Azure:
[PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=955)
Bot commands
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Always choose the latest record for HoodieRecordPayload
> ---
>
> Key: HUDI-2170
> URL: https://issues.apache.org/jira/browse/HUDI-2170
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Common Core
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Now in {{OverwriteWithLatestAvroPayload.preCombine}}, we still choose the old
> record when the new record has the same preCombine field with the old one,
> actually it is more natural to keep the new incoming record instead. The
> {{DefaultHoodieRecordPayload.combineAndGetUpdateValue}} method already did
> that.
> See issue: https://github.com/apache/hudi/issues/3266.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3267: [HUDI-2170] Always choose the latest record for HoodieRecordPayload
hudi-bot edited a comment on pull request #3267: URL: https://github.com/apache/hudi/pull/3267#issuecomment-878894177 ## CI report: * 9c2e6e7b0efc3e235a9c736a095b4e0e003f8b1c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=938) * a275c3af0519430c6ff0dc5902dd84dc5d1a0882 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=955) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2170) Always choose the latest record for HoodieRecordPayload
[
https://issues.apache.org/jira/browse/HUDI-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381748#comment-17381748
]
ASF GitHub Bot commented on HUDI-2170:
--
hudi-bot edited a comment on pull request #3267:
URL: https://github.com/apache/hudi/pull/3267#issuecomment-878894177
## CI report:
* 9c2e6e7b0efc3e235a9c736a095b4e0e003f8b1c Azure:
[FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=938)
* a275c3af0519430c6ff0dc5902dd84dc5d1a0882 UNKNOWN
Bot commands
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Always choose the latest record for HoodieRecordPayload
> ---
>
> Key: HUDI-2170
> URL: https://issues.apache.org/jira/browse/HUDI-2170
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Common Core
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Now in {{OverwriteWithLatestAvroPayload.preCombine}}, we still choose the old
> record when the new record has the same preCombine field with the old one,
> actually it is more natural to keep the new incoming record instead. The
> {{DefaultHoodieRecordPayload.combineAndGetUpdateValue}} method already did
> that.
> See issue: https://github.com/apache/hudi/issues/3266.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3267: [HUDI-2170] Always choose the latest record for HoodieRecordPayload
hudi-bot edited a comment on pull request #3267: URL: https://github.com/apache/hudi/pull/3267#issuecomment-878894177 ## CI report: * 9c2e6e7b0efc3e235a9c736a095b4e0e003f8b1c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=938) * a275c3af0519430c6ff0dc5902dd84dc5d1a0882 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381747#comment-17381747 ] ASF GitHub Bot commented on HUDI-1860: -- Samrat002 commented on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-881168606 @vinothchandar please review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Samrat Deb >Priority: Major > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] Samrat002 commented on pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
Samrat002 commented on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-881168606 @vinothchandar please review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng yunhong reassigned HUDI-1771: --- Assignee: Zheng yunhong (was: Danny Chen) > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381746#comment-17381746 ] ASF GitHub Bot commented on HUDI-1771: -- codecov-commenter edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881152765 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3285](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a517fd1) into [master](https://codecov.io/gh/apache/hudi/commit/a62a6cff32e7d7f62ffdf6362fb8876497fd0528?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a62a6cf) will **decrease** coverage by `20.36%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3285 +/- ## = - Coverage 47.70% 27.33% -20.37% + Complexity 5550 1294 -4256 = Files 936 386 -550 Lines 4162415355-26269 Branches 4188 1341 -2847 = - Hits 19858 4198-15660 + Misses210852 -9148 + Partials 1766 305 -1461 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `20.96% <50.00%> (-13.51%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.23% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUNyZWF0ZUhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...rg/apache/hudi/client/model/HoodieInternalRow.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L21vZGVsL0hvb2RpZUludGVybmFsUm93LmphdmE=) | `79.48% <87.50%> (+1.70%)` | :arrow_up: | | [...java/org/apache/hudi/io/HoodieRowCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca
[hudi] branch master updated (a62a6cf -> 38cd74b)
This is an automated email from the ASF dual-hosted git repository. leesf pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from a62a6cf [MINOR] Refactor hive sync tool to reduce duplicate code (#3276) add 38cd74b [MINOR] Allow users to choose ORC as base file format in Spark SQL (#3279) No new revisions were added by this update. Summary of changes: .../org/apache/hudi/config/HoodieWriteConfig.java | 7 +++ .../org/apache/hudi/HoodieSparkSqlWriter.scala | 2 ++ .../hudi/functional/HoodieSparkSqlWriterSuite.scala | 21 +++-- 3 files changed, 24 insertions(+), 6 deletions(-)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
codecov-commenter edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881152765 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3285](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a517fd1) into [master](https://codecov.io/gh/apache/hudi/commit/a62a6cff32e7d7f62ffdf6362fb8876497fd0528?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a62a6cf) will **decrease** coverage by `20.36%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3285 +/- ## = - Coverage 47.70% 27.33% -20.37% + Complexity 5550 1294 -4256 = Files 936 386 -550 Lines 4162415355-26269 Branches 4188 1341 -2847 = - Hits 19858 4198-15660 + Misses210852 -9148 + Partials 1766 305 -1461 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `20.96% <50.00%> (-13.51%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.23% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3285?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUNyZWF0ZUhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...rg/apache/hudi/client/model/HoodieInternalRow.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L21vZGVsL0hvb2RpZUludGVybmFsUm93LmphdmE=) | `79.48% <87.50%> (+1.70%)` | :arrow_up: | | [...java/org/apache/hudi/io/HoodieRowCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3285/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllUm93Q3JlYXRlSGFuZGxlLmphdmE=) | `92.77% <100.00%> (+0.27%)` | :arrow_up: | | [...main/java/
[GitHub] [hudi] leesf merged pull request #3279: [MINOR] Allow users to choose ORC as base file format in Spark SQL
leesf merged pull request #3279: URL: https://github.com/apache/hudi/pull/3279 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381744#comment-17381744 ] ASF GitHub Bot commented on HUDI-1860: -- codecov-commenter edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870526141 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3184](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (55947cf) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **increase** coverage by `3.69%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3184 +/- ## + Coverage 44.10% 47.80% +3.69% - Complexity 5157 5570 +413 Files 936 936 Lines 4162941632 +3 Branches 4189 4188 -1 + Hits 1836219903+1541 + Misses2163819960-1678 - Partials 1629 1769 +140 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.46% <ø> (ø)` | | | hudicommon | `48.67% <ø> (-0.02%)` | :arrow_down: | | hudiflink | `59.68% <ø> (ø)` | | | hudihadoopmr | `52.02% <ø> (ø)` | | | hudisparkdatasource | `67.35% <ø> (+0.13%)` | :arrow_up: | | hudisync | `55.97% <ø> (+0.24%)` | :arrow_up: | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.37% <100.00%> (+50.25%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.29% <100.00%> (+72.29%)` | :arrow_up: | | [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `78.12% <0.00%> (-1.57%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `78.12% <0.00%> (+0.27%)` | :arrow_up: | | [...in/scala/org/apache/hudi/HoodieStreamingSink.scala](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVN0cmVhbWluZ1Npbmsuc2NhbGE=) | `32.00% <0.00%> (+4.00%)` | :arrow_up: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
codecov-commenter edited a comment on pull request #3184: URL: https://github.com/apache/hudi/pull/3184#issuecomment-870526141 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3184](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (55947cf) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **increase** coverage by `3.69%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3184 +/- ## + Coverage 44.10% 47.80% +3.69% - Complexity 5157 5570 +413 Files 936 936 Lines 4162941632 +3 Branches 4189 4188 -1 + Hits 1836219903+1541 + Misses2163819960-1678 - Partials 1629 1769 +140 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.46% <ø> (ø)` | | | hudicommon | `48.67% <ø> (-0.02%)` | :arrow_down: | | hudiflink | `59.68% <ø> (ø)` | | | hudihadoopmr | `52.02% <ø> (ø)` | | | hudisparkdatasource | `67.35% <ø> (+0.13%)` | :arrow_up: | | hudisync | `55.97% <ø> (+0.24%)` | :arrow_up: | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.37% <100.00%> (+50.25%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `72.29% <100.00%> (+72.29%)` | :arrow_up: | | [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `78.12% <0.00%> (-1.57%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `78.12% <0.00%> (+0.27%)` | :arrow_up: | | [...in/scala/org/apache/hudi/HoodieStreamingSink.scala](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVN0cmVhbWluZ1Npbmsuc2NhbGE=) | `32.00% <0.00%> (+4.00%)` | :arrow_up: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3184/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2R
[jira] [Commented] (HUDI-2145) Create new bucket when NewFileAssignState filled
[ https://issues.apache.org/jira/browse/HUDI-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381743#comment-17381743 ] ASF GitHub Bot commented on HUDI-2145: -- codecov-commenter edited a comment on pull request #3258: URL: https://github.com/apache/hudi/pull/3258#issuecomment-878089095 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3258?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3258](https://codecov.io/gh/apache/hudi/pull/3258?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d2184c0) into [master](https://codecov.io/gh/apache/hudi/commit/5804ad8e32ae05758ebc5e47f5d4fb4db371ab52?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5804ad8) will **increase** coverage by `0.08%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/hudi/pull/3258?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3258 +/- ## + Coverage 47.72% 47.81% +0.08% - Complexity 5528 5569 +41 Files 934 936 +2 Lines 4145741626 +169 Branches 4166 4188 +22 + Hits 1978619902 +116 - Misses1991419957 +43 - Partials 1757 1767 +10 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.46% <ø> (+0.01%)` | :arrow_up: | | hudicommon | `48.69% <ø> (+0.13%)` | :arrow_up: | | hudiflink | `59.73% <100.00%> (-0.31%)` | :arrow_down: | | hudihadoopmr | `52.02% <ø> (+0.73%)` | :arrow_up: | | hudisparkdatasource | `67.41% <ø> (-0.26%)` | :arrow_down: | | hudisync | `55.97% <ø> (+1.46%)` | :arrow_up: | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3258?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...g/apache/hudi/sink/partitioner/BucketAssigner.java](https://codecov.io/gh/apache/hudi/pull/3258/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL0J1Y2tldEFzc2lnbmVyLmphdmE=) | `84.69% <100.00%> (+1.36%)` | :arrow_up: | | [.../java/org/apache/hudi/client/HoodieReadClient.java](https://codecov.io/gh/apache/hudi/pull/3258/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVJlYWRDbGllbnQuamF2YQ==) | `94.64% <0.00%> (-5.36%)` | :arrow_down: | | [...in/scala/org/apache/hudi/HoodieStreamingSink.scala](https://codecov.io/gh/apache/hudi/pull/3258/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVN0cmVhbWluZ1Npbmsuc2NhbGE=) | `33.60% <0.00%> (-4.80%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/util/StreamerUtil.java](https://codecov.io/gh/apache/hudi/pull/3258/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1N0cmVhbWVyVXRpbC5qYXZh) | `64.00% <0.00%> (-3.80%)` | :arrow_down: | | [.../hudi/common/util/collection/LazyFileIterable.java](https:/
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3258: [HUDI-2145] Create new bucket when NewFileAssignState filled
codecov-commenter edited a comment on pull request #3258: URL: https://github.com/apache/hudi/pull/3258#issuecomment-878089095 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3258?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3258](https://codecov.io/gh/apache/hudi/pull/3258?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d2184c0) into [master](https://codecov.io/gh/apache/hudi/commit/5804ad8e32ae05758ebc5e47f5d4fb4db371ab52?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5804ad8) will **increase** coverage by `0.08%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/hudi/pull/3258?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3258 +/- ## + Coverage 47.72% 47.81% +0.08% - Complexity 5528 5569 +41 Files 934 936 +2 Lines 4145741626 +169 Branches 4166 4188 +22 + Hits 1978619902 +116 - Misses1991419957 +43 - Partials 1757 1767 +10 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.46% <ø> (+0.01%)` | :arrow_up: | | hudicommon | `48.69% <ø> (+0.13%)` | :arrow_up: | | hudiflink | `59.73% <100.00%> (-0.31%)` | :arrow_down: | | hudihadoopmr | `52.02% <ø> (+0.73%)` | :arrow_up: | | hudisparkdatasource | `67.41% <ø> (-0.26%)` | :arrow_down: | | hudisync | `55.97% <ø> (+1.46%)` | :arrow_up: | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3258?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...g/apache/hudi/sink/partitioner/BucketAssigner.java](https://codecov.io/gh/apache/hudi/pull/3258/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL0J1Y2tldEFzc2lnbmVyLmphdmE=) | `84.69% <100.00%> (+1.36%)` | :arrow_up: | | [.../java/org/apache/hudi/client/HoodieReadClient.java](https://codecov.io/gh/apache/hudi/pull/3258/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVJlYWRDbGllbnQuamF2YQ==) | `94.64% <0.00%> (-5.36%)` | :arrow_down: | | [...in/scala/org/apache/hudi/HoodieStreamingSink.scala](https://codecov.io/gh/apache/hudi/pull/3258/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVN0cmVhbWluZ1Npbmsuc2NhbGE=) | `33.60% <0.00%> (-4.80%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/util/StreamerUtil.java](https://codecov.io/gh/apache/hudi/pull/3258/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1N0cmVhbWVyVXRpbC5qYXZh) | `64.00% <0.00%> (-3.80%)` | :arrow_down: | | [.../hudi/common/util/collection/LazyFileIterable.java](https://codecov.io/gh/apache/hudi/pull/3258/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V
[jira] [Commented] (HUDI-1633) Make callback return HoodieWriteStat
[ https://issues.apache.org/jira/browse/HUDI-1633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381740#comment-17381740 ] ASF GitHub Bot commented on HUDI-1633: -- codecov-commenter edited a comment on pull request #2445: URL: https://github.com/apache/hudi/pull/2445#issuecomment-881149916 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2445?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2445](https://codecov.io/gh/apache/hudi/pull/2445?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1068b7f) into [master](https://codecov.io/gh/apache/hudi/commit/e3d3677b7e7899705b624925666317f0c074f7c7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e3d3677) will **increase** coverage by `4.53%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/2445?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2445 +/- ## + Coverage 50.73% 55.26% +4.53% - Complexity 3064 4364+1300 Files 419 548 +129 Lines 1879727555+8758 Branches 1922 3093+1171 + Hits 953615229+5693 - Misses 848710863+2376 - Partials774 1463 +689 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (+2.71%)` | :arrow_up: | | hudiclient | `∅ <ø> (∅)` | | | hudicommon | `49.31% <ø> (-2.70%)` | :arrow_down: | | hudiflink | `59.68% <ø> (+49.48%)` | :arrow_up: | | hudihadoopmr | `52.08% <ø> (+19.01%)` | :arrow_up: | | hudisparkdatasource | `67.21% <ø> (+1.31%)` | :arrow_up: | | hudisync | `55.97% <ø> (+7.36%)` | :arrow_up: | | huditimelineservice | `64.07% <ø> (-2.77%)` | :arrow_down: | | hudiutilities | `72.84% <0.00%> (+3.36%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2445?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...callback/kafka/HoodieWriteCommitKafkaCallback.java](https://codecov.io/gh/apache/hudi/pull/2445/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NhbGxiYWNrL2thZmthL0hvb2RpZVdyaXRlQ29tbWl0S2Fma2FDYWxsYmFjay5qYXZh) | `0.00% <0.00%> (ø)` | | | [...e/hudi/common/util/collection/RocksDBBasedMap.java](https://codecov.io/gh/apache/hudi/pull/2445/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvY29sbGVjdGlvbi9Sb2Nrc0RCQmFzZWRNYXAuamF2YQ==) | `0.00% <0.00%> (-39.29%)` | :arrow_down: | | [.../java/org/apache/hudi/common/util/CommitUtils.java](https://codecov.io/gh/apache/hudi/pull/2445/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ29tbWl0VXRpbHMuamF2YQ==) | `40.47% <0.00%> (-31.53%)` | :arrow_down: | | [...g/apache/hudi/utilities/schema/SchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2445/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlci5qYXZh) | `71.42% <0.00%> (-28.58%)` | :arrow_down: | | [...java/org/apache/hud
