[GitHub] [hudi] hudi-bot edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
hudi-bot edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867438049 ## CI report: * 9f29a8afc1a6e049984a789001042921d917f180 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=431) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3120?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3120](https://codecov.io/gh/apache/hudi/pull/3120?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e1d659e) into [master](https://codecov.io/gh/apache/hudi/commit/0fb8556b0d9274aef650a46bb82a8cf495d4450b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0fb8556) will **decrease** coverage by `15.95%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3120?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3120 +/- ## = - Coverage 44.28% 28.32% -15.96% + Complexity 4573 1255 -3318 = Files 824 375 -449 Lines 3639114346-22045 Branches 3939 1467 -2472 = - Hits 16117 4064-12053 + Misses18528 9986 -8542 + Partials 1746 296 -1450 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (+5.84%)` | :arrow_up: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.18% <0.00%> (-48.87%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.29% <ø> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3120?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...c/main/java/org/apache/hudi/hive/HiveSyncTool.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNUb29sLmphdmE=) | `0.00% <0.00%> (-72.36%)` | :arrow_down: | | [...in/java/org/apache/hudi/hive/util/ConfigUtils.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9Db25maWdVdGlscy5qYXZh) | `0.00% <ø> (-73.92%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3120/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2963: [HUDI-1904] Introduce SchemaProviderInterface to make SchemaProvider unified
codecov-commenter edited a comment on pull request #2963: URL: https://github.com/apache/hudi/pull/2963#issuecomment-843155329 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2963](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ce253fe) into [master](https://codecov.io/gh/apache/hudi/commit/08464a6a5b11ad2afa98eed7bc4ec4121b6b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (08464a6) will **increase** coverage by `0.90%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2963 +/- ## + Coverage 49.86% 50.77% +0.90% + Complexity 3527 443-3084 Files 488 77 -411 Lines 23618 3234 -20384 Branches 2528 396-2132 - Hits 11777 1642 -10135 + Misses10802 1439-9363 + Partials 1039 153 -886 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `∅ <ø> (∅)` | | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-41.22%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `72.28% <ø> (+63.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...g/apache/hudi/utilities/schema/SchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlci5qYXZh) | `66.66% <ø> (+9.52%)` | :arrow_up: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [.../apache/hudi/hive/MultiPartKeysValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTXVsdGlQYXJ0S2V5c1ZhbHVlRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (-90.91%)` | :arrow_down: | | [...in/java
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2963: [HUDI-1904] Introduce SchemaProviderInterface to make SchemaProvider unified
codecov-commenter edited a comment on pull request #2963: URL: https://github.com/apache/hudi/pull/2963#issuecomment-843155329 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2963](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ce253fe) into [master](https://codecov.io/gh/apache/hudi/commit/08464a6a5b11ad2afa98eed7bc4ec4121b6b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (08464a6) will **increase** coverage by `0.45%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2963 +/- ## + Coverage 49.86% 50.32% +0.45% - Complexity 3527 3543 +16 Files 488 462 -26 Lines 2361822176-1442 Branches 2528 2369 -159 - Hits 1177711159 -618 + Misses1080210063 -739 + Partials 1039 954 -85 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (+0.40%)` | :arrow_up: | | hudiclient | `∅ <ø> (∅)` | | | hudicommon | `48.21% <ø> (-2.08%)` | :arrow_down: | | hudiflink | `61.49% <ø> (-1.78%)` | :arrow_down: | | hudihadoopmr | `51.34% <ø> (-0.09%)` | :arrow_down: | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-41.22%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `72.28% <ø> (+63.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/schema/FilebasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zY2hlbWEvRmlsZWJhc2VkU2NoZW1hUHJvdmlkZXIuamF2YQ==) | `77.77% <ø> (+48.61%)` | :arrow_up: | | [...g/apache/hudi/utilities/schema/SchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlci5qYXZh) | `66.66% <ø> (+9.52%)` | :arrow_up: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hp
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
codecov-commenter edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864770805 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2963: [HUDI-1904] Introduce SchemaProviderInterface to make SchemaProvider unified
codecov-commenter edited a comment on pull request #2963: URL: https://github.com/apache/hudi/pull/2963#issuecomment-843155329 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2963](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ce253fe) into [master](https://codecov.io/gh/apache/hudi/commit/08464a6a5b11ad2afa98eed7bc4ec4121b6b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (08464a6) will **increase** coverage by `4.88%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2963 +/- ## + Coverage 49.86% 54.74% +4.88% - Complexity 3527 4168 +641 Files 488 540 +52 Lines 2361826538+2920 Branches 2528 3039 +511 + Hits 1177714529+2752 + Misses1080210569 -233 - Partials 1039 1440 +401 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (+0.40%)` | :arrow_up: | | hudiclient | `∅ <ø> (∅)` | | | hudicommon | `48.21% <ø> (-2.08%)` | :arrow_down: | | hudiflink | `61.49% <ø> (-1.78%)` | :arrow_down: | | hudihadoopmr | `51.34% <ø> (-0.09%)` | :arrow_down: | | hudisparkdatasource | `67.06% <ø> (-7.24%)` | :arrow_down: | | hudisync | `54.05% <ø> (+7.45%)` | :arrow_up: | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `72.28% <ø> (+63.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2963?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/schema/FilebasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zY2hlbWEvRmlsZWJhc2VkU2NoZW1hUHJvdmlkZXIuamF2YQ==) | `77.77% <ø> (+48.61%)` | :arrow_up: | | [...g/apache/hudi/utilities/schema/SchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlci5qYXZh) | `66.66% <ø> (+9.52%)` | :arrow_up: | | [.../org/apache/hudi/sink/compact/CompactFunction.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdEZ1bmN0aW9uLmphdmE=) | `86.66% <0.00%> (-13.34%)` | :arrow_down: | | [...e/hudi/sink/partitioner/profile/WriteProfiles.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL3Byb2ZpbGUvV3JpdGVQcm9maWxlcy5qYXZh) | `46.80% <0.00%> (-13.20%)` | :arrow_down: | | [...apache/hudi/sink/compact/CompactionCommitSink.java](https://codecov.io/gh/apache/hudi/pull/2963/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ2
[GitHub] [hudi] wangxianghu opened a new pull request #3152: [MINOR] Remove unused methods in StreamUtil
wangxianghu opened a new pull request #3152: URL: https://github.com/apache/hudi/pull/3152 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 88bb69cab32a8faa94703c211007f526582ffbe2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=432) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3152: [MINOR] Remove unused methods in StreamUtil
hudi-bot commented on pull request #3152: URL: https://github.com/apache/hudi/pull/3152#issuecomment-868282420 ## CI report: * efb28c1b1d781d5d384bbf8ed450feef87d43e6b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2072) Add Precommit validator framework
[ https://issues.apache.org/jira/browse/HUDI-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2072: - Labels: pull-request-available (was: ) > Add Precommit validator framework > - > > Key: HUDI-2072 > URL: https://issues.apache.org/jira/browse/HUDI-2072 > Project: Apache Hudi > Issue Type: New Feature >Reporter: satish >Assignee: satish >Priority: Major > Labels: pull-request-available > > We want to run pre-commit validators before 'promoting' a inflight operation > to commit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] satishkotha opened a new pull request #3153: [HUDI-2072] Add pre-commit validator framework
satishkotha opened a new pull request #3153: URL: https://github.com/apache/hudi/pull/3153 ## What is the purpose of the pull request Add pre-commit validator framework to run queries and compare query results ## Brief change log * Add pre-commit validator for spark * Add sql query based validations to compare partition state a) before the commit started AND b) if commit were to complete. ## Verify this pull request This change added tests ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] satishkotha commented on pull request #3153: [HUDI-2072] Add pre-commit validator framework
satishkotha commented on pull request #3153: URL: https://github.com/apache/hudi/pull/3153#issuecomment-868284703 @bvaradar PTAL and let me know if this is useful for you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
codecov-commenter edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867510590 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3146](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9f29a8a) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `43.16%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3146 +/- ## - Coverage 46.24% 3.07% -43.17% + Complexity 5362 82 -5280 Files 920 279 -641 Lines 39842 10861-28981 Branches 42911108 -3183 - Hits 18425 334-18091 + Misses19535 10501 -9034 + Partials 1882 26 -1856 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.35% <0.00%> (-48.70%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.52% <ø> (-48.77%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...java/org/apache/hudi/hive/util/HiveSchemaUtil.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlU2NoZW1hVXRpbC5qYXZh) | `0.00% <0.00%> (-68.94%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/so
[GitHub] [hudi] hudi-bot edited a comment on pull request #3152: [MINOR] Remove unused methods in StreamUtil
hudi-bot edited a comment on pull request #3152: URL: https://github.com/apache/hudi/pull/3152#issuecomment-868282420 ## CI report: * efb28c1b1d781d5d384bbf8ed450feef87d43e6b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=434) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3153: [HUDI-2072] Add pre-commit validator framework
hudi-bot commented on pull request #3153: URL: https://github.com/apache/hudi/pull/3153#issuecomment-868285452 ## CI report: * 879371e767586a6229f24da8bcb469a59b7c6893 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3153: [HUDI-2072] Add pre-commit validator framework
hudi-bot edited a comment on pull request #3153: URL: https://github.com/apache/hudi/pull/3153#issuecomment-868285452 ## CI report: * 879371e767586a6229f24da8bcb469a59b7c6893 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=435) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * e1d659e1b04a7b262d7ee65bc5706e2d355ca5a3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=433) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
codecov-commenter edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867510590 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3146](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9f29a8a) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `30.06%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3146 +/- ## = - Coverage 46.24% 16.17% -30.07% + Complexity 5362 467 -4895 = Files 920 279 -641 Lines 3984210861-28981 Branches 4291 1108 -3183 = - Hits 18425 1757-16668 + Misses19535 8945-10590 + Partials 1882 159 -1723 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.35% <0.00%> (-48.70%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.29% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...java/org/apache/hudi/hive/util/HiveSchemaUtil.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlU2NoZW1hVXRpbC5qYXZh) | `0.00% <0.00%> (-68.94%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
[GitHub] [hudi] codecov-commenter commented on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
codecov-commenter commented on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868296609 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3151](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (88bb69c) into [master](https://codecov.io/gh/apache/hudi/commit/01ad449ad68d9c77ee1493fcdf833df53df6106a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01ad449) will **decrease** coverage by `2.43%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3151 +/- ## - Coverage 52.19% 49.76% -2.44% + Complexity 2659 393-2266 Files 335 66 -269 Lines 14981 2918 -12063 Branches 1505 319-1186 - Hits 7819 1452-6367 + Misses 6536 1329-5207 + Partials626 137 -489 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `?` | | | hudicommon | `?` | | | hudihadoopmr | `?` | | | huditimelineservice | `?` | | | hudiutilities | `49.76% <ø> (-19.90%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.53%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | :arrow_down: | | [...g/apache/hudi/utilities/schema/SchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlci5qYXZh) | `71.42% <0.00%> (-28.58%)` | :arrow_down: | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=re
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
codecov-commenter edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867510590 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3146](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9f29a8a) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `17.84%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3146 +/- ## = - Coverage 46.24% 28.39% -17.85% + Complexity 5362 1255 -4107 = Files 920 375 -545 Lines 3984214312-25530 Branches 4291 1460 -2831 = - Hits 18425 4064-14361 + Misses19535 9952 -9583 + Partials 1882 296 -1586 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.35% <0.00%> (-48.70%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.29% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `0.00% <0.00%> (-97.83%)` | :arrow_down: | | [...java/org/apache/hudi/hive/util/HiveSchemaUtil.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlU2NoZW1hVXRpbC5qYXZh) | `0.00% <0.00%> (-68.94%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
codecov-commenter edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868296609 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3151](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (88bb69c) into [master](https://codecov.io/gh/apache/hudi/commit/01ad449ad68d9c77ee1493fcdf833df53df6106a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01ad449) will **increase** coverage by `6.82%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3151 +/- ## + Coverage 52.19% 59.02% +6.82% + Complexity 2659 1181-1478 Files 335 162 -173 Lines 14981 6369-8612 Branches 1505 671 -834 - Hits 7819 3759-4060 + Misses 6536 2336-4200 + Partials626 274 -352 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `66.85% <ø> (∅)` | | | hudicommon | `?` | | | hudihadoopmr | `?` | | | huditimelineservice | `?` | | | hudiutilities | `49.76% <ø> (-19.90%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...rg/apache/hudi/async/SparkAsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYXN5bmMvU3BhcmtBc3luY0NvbXBhY3RTZXJ2aWNlLmphdmE=) | `0.00% <ø> (ø)` | | | [.../java/org/apache/hudi/client/HoodieReadClient.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVJlYWRDbGllbnQuamF2YQ==) | `100.00% <ø> (ø)` | | | [...a/org/apache/hudi/client/HoodieSparkCompactor.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVNwYXJrQ29tcGFjdG9yLmphdmE=) | `0.00% <ø> (ø)` | | | [...va/org/apache/hudi/client/SparkRDDWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrUkREV3JpdGVDbGllbnQuamF2YQ==) | `72.05% <ø> (ø)` | | | [...g/apache/hudi/client/SparkTaskContextSupplier.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrVGFza0NvbnRleHRTdXBwbGllci5qYXZh) | `89.47% <ø> (ø)` | | | [.../bootstrap/HoodieSparkBootstrapSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
codecov-commenter edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867510590 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3146](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9f29a8a) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `13.70%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3146 +/- ## = - Coverage 46.24% 32.54% -13.71% + Complexity 5362 1439 -3923 = Files 920 379 -541 Lines 3984214691-25151 Branches 4291 1489 -2802 = - Hits 18425 4781-13644 + Misses19535 9540 -9995 + Partials 1882 370 -1512 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `54.24% <100.00%> (+0.19%)` | :arrow_up: | | huditimelineservice | `?` | | | hudiutilities | `58.29% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `97.95% <100.00%> (+0.13%)` | :arrow_up: | | [...java/org/apache/hudi/hive/util/HiveSchemaUtil.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlU2NoZW1hVXRpbC5qYXZh) | `69.37% <100.00%> (+0.44%)` | :arrow_up: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_do
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3146: [HUDI-2057] CTAS Generate An External Table When Create Managed Table
codecov-commenter edited a comment on pull request #3146: URL: https://github.com/apache/hudi/pull/3146#issuecomment-867510590 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3146](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9f29a8a) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **increase** coverage by `0.01%`. > The diff coverage is `90.90%`. [](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3146 +/- ## + Coverage 46.24% 46.26% +0.01% - Complexity 5362 5364 +2 Files 920 920 Lines 3984239852 +10 Branches 4291 4293 +2 + Hits 1842518436 +11 + Misses1953519534 -1 Partials 1882 1882 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <ø> (ø)` | | | hudiclient | `30.45% <ø> (ø)` | | | hudicommon | `47.58% <ø> (+0.01%)` | :arrow_up: | | hudiflink | `61.25% <ø> (ø)` | | | hudihadoopmr | `51.29% <ø> (ø)` | | | hudisparkdatasource | `67.07% <75.00%> (+<0.01%)` | :arrow_up: | | hudisync | `54.24% <100.00%> (+0.19%)` | :arrow_up: | | huditimelineservice | `64.36% <ø> (ø)` | | | hudiutilities | `58.29% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...udi/command/CreateHoodieTableAsSelectCommand.scala](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvaHVkaS9jb21tYW5kL0NyZWF0ZUhvb2RpZVRhYmxlQXNTZWxlY3RDb21tYW5kLnNjYWxh) | `63.82% <50.00%> (-0.62%)` | :arrow_down: | | [...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVNwYXJrU3FsV3JpdGVyLnNjYWxh) | `70.98% <100.00%> (+0.16%)` | :arrow_up: | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | `97.95% <100.00%> (+0.13%)` | :arrow_up: | | [...java/org/apache/hudi/hive/util/HiveSchemaUtil.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlU2NoZW1hVXRpbC5qYXZh) | `69.37% <100.00%> (+0.44%)` | :arrow_up: | | [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/3146/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvY
[GitHub] [hudi] hudi-bot edited a comment on pull request #3152: [MINOR] Remove unused methods in StreamUtil
hudi-bot edited a comment on pull request #3152: URL: https://github.com/apache/hudi/pull/3152#issuecomment-868282420 ## CI report: * efb28c1b1d781d5d384bbf8ed450feef87d43e6b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=434) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3153: [HUDI-2072] Add pre-commit validator framework
hudi-bot edited a comment on pull request #3153: URL: https://github.com/apache/hudi/pull/3153#issuecomment-868285452 ## CI report: * 879371e767586a6229f24da8bcb469a59b7c6893 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=435) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
yanghua commented on a change in pull request #3151:
URL: https://github.com/apache/hudi/pull/3151#discussion_r658572030
##
File path:
hudi-flink/src/main/java/org/apache/hudi/streamer/FlinkStreamerConfig.java
##
@@ -130,6 +131,183 @@
@Parameter(names = {"--write-task-num"}, description = "Parallelism of tasks
that do actual write, default is 4.")
public Integer writeTaskNum = 4;
+ @Parameter(names = {"--partition-default-name"},
+ description = "The default partition name in case the dynamic
partition column value is null/empty string")
+ public String partitionDefaultName = "__DEFAULT_PARTITION__";
+
+ @Parameter(names = {"--index-bootstrap-enabled"},
+ description = "Whether to bootstrap the index state from existing
hoodie table, default false")
+ public Boolean indexBootstrapEnabled = false;
+
+ @Parameter(names = {"--index-state-ttl"}, description = "Index state ttl in
days, default 1.5 day")
+ public Double indexStateTtl = 1.5D;
+
+ @Parameter(names = {"--index-global-enabled"}, description = "Whether to
update index for the old partition path " +
+ "if same key record with different partition path came in, default
false")
+ public Boolean indexGlobalEnabled = false;
+
+ @Parameter(names = {"--index-partition-regex"},
+ description = "Whether to load partitions in state if partition path
matching， default *")
+ public String indexPartitionRegex = ".*";
+
+ @Parameter(names = {"--read-tasks-parallelism"}, description = "Parallelism
of tasks that do actual read, default is 4")
+ public Integer readTasksParallelism = 4;
+
+ @Parameter(names = {"--avro-schema-path"}, description = "Avro schema file
path, the parsed schema is used for deserialization")
+ public String avroSchemaPath;
+
+ @Parameter(names = {"--avro-schema"}, description = "Avro schema string, the
parsed schema is used for deserialization")
+ public String avroSchema;
Review comment:
It has no default value, and is not required. It would cause NPE when
copying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
yanghua commented on a change in pull request #3151:
URL: https://github.com/apache/hudi/pull/3151#discussion_r658572030
##
File path:
hudi-flink/src/main/java/org/apache/hudi/streamer/FlinkStreamerConfig.java
##
@@ -130,6 +131,183 @@
@Parameter(names = {"--write-task-num"}, description = "Parallelism of tasks
that do actual write, default is 4.")
public Integer writeTaskNum = 4;
+ @Parameter(names = {"--partition-default-name"},
+ description = "The default partition name in case the dynamic
partition column value is null/empty string")
+ public String partitionDefaultName = "__DEFAULT_PARTITION__";
+
+ @Parameter(names = {"--index-bootstrap-enabled"},
+ description = "Whether to bootstrap the index state from existing
hoodie table, default false")
+ public Boolean indexBootstrapEnabled = false;
+
+ @Parameter(names = {"--index-state-ttl"}, description = "Index state ttl in
days, default 1.5 day")
+ public Double indexStateTtl = 1.5D;
+
+ @Parameter(names = {"--index-global-enabled"}, description = "Whether to
update index for the old partition path " +
+ "if same key record with different partition path came in, default
false")
+ public Boolean indexGlobalEnabled = false;
+
+ @Parameter(names = {"--index-partition-regex"},
+ description = "Whether to load partitions in state if partition path
matching， default *")
+ public String indexPartitionRegex = ".*";
+
+ @Parameter(names = {"--read-tasks-parallelism"}, description = "Parallelism
of tasks that do actual read, default is 4")
+ public Integer readTasksParallelism = 4;
+
+ @Parameter(names = {"--avro-schema-path"}, description = "Avro schema file
path, the parsed schema is used for deserialization")
+ public String avroSchemaPath;
+
+ @Parameter(names = {"--avro-schema"}, description = "Avro schema string, the
parsed schema is used for deserialization")
+ public String avroSchema;
Review comment:
It has no default value, and is not required. It would cause exception
when copying if we do not verify before setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
danny0405 commented on a change in pull request #3151:
URL: https://github.com/apache/hudi/pull/3151#discussion_r658574590
##
File path:
hudi-flink/src/main/java/org/apache/hudi/streamer/FlinkStreamerConfig.java
##
@@ -130,6 +131,183 @@
@Parameter(names = {"--write-task-num"}, description = "Parallelism of tasks
that do actual write, default is 4.")
public Integer writeTaskNum = 4;
+ @Parameter(names = {"--partition-default-name"},
+ description = "The default partition name in case the dynamic
partition column value is null/empty string")
+ public String partitionDefaultName = "__DEFAULT_PARTITION__";
+
+ @Parameter(names = {"--index-bootstrap-enabled"},
+ description = "Whether to bootstrap the index state from existing
hoodie table, default false")
+ public Boolean indexBootstrapEnabled = false;
+
+ @Parameter(names = {"--index-state-ttl"}, description = "Index state ttl in
days, default 1.5 day")
+ public Double indexStateTtl = 1.5D;
+
+ @Parameter(names = {"--index-global-enabled"}, description = "Whether to
update index for the old partition path " +
+ "if same key record with different partition path came in, default
false")
+ public Boolean indexGlobalEnabled = false;
+
+ @Parameter(names = {"--index-partition-regex"},
+ description = "Whether to load partitions in state if partition path
matching， default *")
+ public String indexPartitionRegex = ".*";
+
+ @Parameter(names = {"--read-tasks-parallelism"}, description = "Parallelism
of tasks that do actual read, default is 4")
+ public Integer readTasksParallelism = 4;
+
+ @Parameter(names = {"--avro-schema-path"}, description = "Avro schema file
path, the parsed schema is used for deserialization")
+ public String avroSchemaPath;
+
+ @Parameter(names = {"--avro-schema"}, description = "Avro schema string, the
parsed schema is used for deserialization")
+ public String avroSchema;
+
+ @Parameter(names = {"--source-query-type"}, description = "Decides how data
files need to be read, in\n"
+ + "1) Snapshot mode (obtain latest view, based on row & columnar
data);\n"
+ + "2) incremental mode (new data since an instantTime);\n"
+ + "3) Read Optimized mode (obtain latest view, based on columnar
data)\n."
+ + "Default: snapshot")
Review comment:
I guess we do not need these read options, because the
`HoodieFlinkStreamer` only do write operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
veenaypatil commented on a change in pull request #3151:
URL: https://github.com/apache/hudi/pull/3151#discussion_r658588071
##
File path:
hudi-flink/src/main/java/org/apache/hudi/streamer/FlinkStreamerConfig.java
##
@@ -130,6 +131,183 @@
@Parameter(names = {"--write-task-num"}, description = "Parallelism of tasks
that do actual write, default is 4.")
public Integer writeTaskNum = 4;
+ @Parameter(names = {"--partition-default-name"},
+ description = "The default partition name in case the dynamic
partition column value is null/empty string")
+ public String partitionDefaultName = "__DEFAULT_PARTITION__";
+
+ @Parameter(names = {"--index-bootstrap-enabled"},
+ description = "Whether to bootstrap the index state from existing
hoodie table, default false")
+ public Boolean indexBootstrapEnabled = false;
+
+ @Parameter(names = {"--index-state-ttl"}, description = "Index state ttl in
days, default 1.5 day")
+ public Double indexStateTtl = 1.5D;
+
+ @Parameter(names = {"--index-global-enabled"}, description = "Whether to
update index for the old partition path " +
+ "if same key record with different partition path came in, default
false")
+ public Boolean indexGlobalEnabled = false;
+
+ @Parameter(names = {"--index-partition-regex"},
+ description = "Whether to load partitions in state if partition path
matching， default *")
+ public String indexPartitionRegex = ".*";
+
+ @Parameter(names = {"--read-tasks-parallelism"}, description = "Parallelism
of tasks that do actual read, default is 4")
+ public Integer readTasksParallelism = 4;
+
+ @Parameter(names = {"--avro-schema-path"}, description = "Avro schema file
path, the parsed schema is used for deserialization")
+ public String avroSchemaPath;
+
+ @Parameter(names = {"--avro-schema"}, description = "Avro schema string, the
parsed schema is used for deserialization")
+ public String avroSchema;
Review comment:
adding empty string as default value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
veenaypatil commented on a change in pull request #3151:
URL: https://github.com/apache/hudi/pull/3151#discussion_r658588403
##
File path:
hudi-flink/src/main/java/org/apache/hudi/streamer/FlinkStreamerConfig.java
##
@@ -130,6 +131,183 @@
@Parameter(names = {"--write-task-num"}, description = "Parallelism of tasks
that do actual write, default is 4.")
public Integer writeTaskNum = 4;
+ @Parameter(names = {"--partition-default-name"},
+ description = "The default partition name in case the dynamic
partition column value is null/empty string")
+ public String partitionDefaultName = "__DEFAULT_PARTITION__";
+
+ @Parameter(names = {"--index-bootstrap-enabled"},
+ description = "Whether to bootstrap the index state from existing
hoodie table, default false")
+ public Boolean indexBootstrapEnabled = false;
+
+ @Parameter(names = {"--index-state-ttl"}, description = "Index state ttl in
days, default 1.5 day")
+ public Double indexStateTtl = 1.5D;
+
+ @Parameter(names = {"--index-global-enabled"}, description = "Whether to
update index for the old partition path " +
+ "if same key record with different partition path came in, default
false")
+ public Boolean indexGlobalEnabled = false;
+
+ @Parameter(names = {"--index-partition-regex"},
+ description = "Whether to load partitions in state if partition path
matching， default *")
+ public String indexPartitionRegex = ".*";
+
+ @Parameter(names = {"--read-tasks-parallelism"}, description = "Parallelism
of tasks that do actual read, default is 4")
+ public Integer readTasksParallelism = 4;
+
+ @Parameter(names = {"--avro-schema-path"}, description = "Avro schema file
path, the parsed schema is used for deserialization")
+ public String avroSchemaPath;
+
+ @Parameter(names = {"--avro-schema"}, description = "Avro schema string, the
parsed schema is used for deserialization")
+ public String avroSchema;
+
+ @Parameter(names = {"--source-query-type"}, description = "Decides how data
files need to be read, in\n"
+ + "1) Snapshot mode (obtain latest view, based on row & columnar
data);\n"
+ + "2) incremental mode (new data since an instantTime);\n"
+ + "3) Read Optimized mode (obtain latest view, based on columnar
data)\n."
+ + "Default: snapshot")
Review comment:
removed
```
source-query-type
source-merge-type
source-hive-style-partition
read-tasks-parallelism
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 88bb69cab32a8faa94703c211007f526582ffbe2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=432) * 15cb914ed60ef660fdd4690eb80d2c6bd880f199 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
danny0405 commented on a change in pull request #3151:
URL: https://github.com/apache/hudi/pull/3151#discussion_r658597164
##
File path:
hudi-flink/src/main/java/org/apache/hudi/streamer/FlinkStreamerConfig.java
##
@@ -130,6 +131,162 @@
@Parameter(names = {"--write-task-num"}, description = "Parallelism of tasks
that do actual write, default is 4.")
public Integer writeTaskNum = 4;
+ @Parameter(names = {"--partition-default-name"},
+ description = "The default partition name in case the dynamic
partition column value is null/empty string")
+ public String partitionDefaultName = "__DEFAULT_PARTITION__";
+
+ @Parameter(names = {"--index-bootstrap-enabled"},
+ description = "Whether to bootstrap the index state from existing
hoodie table, default false")
+ public Boolean indexBootstrapEnabled = false;
+
+ @Parameter(names = {"--index-state-ttl"}, description = "Index state ttl in
days, default 1.5 day")
+ public Double indexStateTtl = 1.5D;
+
+ @Parameter(names = {"--index-global-enabled"}, description = "Whether to
update index for the old partition path "
+ + "if same key record with different partition path came in, default
false")
+ public Boolean indexGlobalEnabled = false;
+
+ @Parameter(names = {"--index-partition-regex"},
+ description = "Whether to load partitions in state if partition path
matching， default *")
+ public String indexPartitionRegex = ".*";
+
+ @Parameter(names = {"--avro-schema-path"}, description = "Avro schema file
path, the parsed schema is used for deserialization")
+ public String avroSchemaPath = "";
+
+ @Parameter(names = {"--avro-schema"}, description = "Avro schema string, the
parsed schema is used for deserialization")
+ public String avroSchema = "";
+
+ @Parameter(names = {"--utc-timezone"}, description = "Use UTC timezone or
local timezone to the conversion between epoch"
+ + " time and LocalDateTime. Hive 0.x/1.x/2.x use local timezone. But
Hive 3.x"
+ + " use UTC timezone, by default true")
+ public Boolean utcTimezone = true;
+
Review comment:
The following 3 are also read options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 88bb69cab32a8faa94703c211007f526582ffbe2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=432) * 15cb914ed60ef660fdd4690eb80d2c6bd880f199 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=436) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a change in pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
yuzhaojing commented on a change in pull request #3134:
URL: https://github.com/apache/hudi/pull/3134#discussion_r658609372
##
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapFunction.java
##
@@ -168,32 +175,69 @@ private HoodieFlinkTable getTable() {
* @param partitionPath The partition path
*/
@SuppressWarnings("unchecked")
- private void loadRecords(String partitionPath, Collector out) {
+ private void loadRecords(String partitionPath, Collector out) throws
Exception {
long start = System.currentTimeMillis();
+
BaseFileUtils fileUtils =
BaseFileUtils.getInstance(this.hoodieTable.getBaseFileFormat());
-List latestBaseFiles =
-HoodieIndexUtils.getLatestBaseFilesForPartition(partitionPath,
this.hoodieTable);
-LOG.info("All baseFile in partition {} size = {}", partitionPath,
latestBaseFiles.size());
+Schema schema = new
TableSchemaResolver(this.hoodieTable.getMetaClient()).getTableAvroSchema();
final int parallelism = getRuntimeContext().getNumberOfParallelSubtasks();
final int maxParallelism =
getRuntimeContext().getMaxNumberOfParallelSubtasks();
final int taskID = getRuntimeContext().getIndexOfThisSubtask();
-for (HoodieBaseFile baseFile : latestBaseFiles) {
- boolean shouldLoad = KeyGroupRangeAssignment.assignKeyToParallelOperator(
- baseFile.getFileId(), maxParallelism, parallelism) == taskID;
- if (shouldLoad) {
-LOG.info("Load records from file {}.", baseFile);
-final List hoodieKeys;
+Option latestCommitTime =
this.hoodieTable.getMetaClient().getCommitsTimeline()
+.filterCompletedInstants().lastInstant();
+
+if (latestCommitTime.isPresent()) {
+ List fileSlices = this.hoodieTable.getSliceView()
+ .getLatestFileSlicesBeforeOrOn(partitionPath,
latestCommitTime.get().getTimestamp(), true)
+ .collect(toList());
+
+ for (FileSlice fileSlice : fileSlices) {
+if (!shouldLoadFileId(fileSlice.getFileId(), maxParallelism,
parallelism, taskID)) {
+ return;
Review comment:
Thanks for review, I will fix that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 88bb69cab32a8faa94703c211007f526582ffbe2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=432) * 15cb914ed60ef660fdd4690eb80d2c6bd880f199 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=436) * 90781844a82336b56dbf32a39cf9d65874c5a57e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
veenaypatil commented on a change in pull request #3151:
URL: https://github.com/apache/hudi/pull/3151#discussion_r658612045
##
File path:
hudi-flink/src/main/java/org/apache/hudi/streamer/FlinkStreamerConfig.java
##
@@ -130,6 +131,162 @@
@Parameter(names = {"--write-task-num"}, description = "Parallelism of tasks
that do actual write, default is 4.")
public Integer writeTaskNum = 4;
+ @Parameter(names = {"--partition-default-name"},
+ description = "The default partition name in case the dynamic
partition column value is null/empty string")
+ public String partitionDefaultName = "__DEFAULT_PARTITION__";
+
+ @Parameter(names = {"--index-bootstrap-enabled"},
+ description = "Whether to bootstrap the index state from existing
hoodie table, default false")
+ public Boolean indexBootstrapEnabled = false;
+
+ @Parameter(names = {"--index-state-ttl"}, description = "Index state ttl in
days, default 1.5 day")
+ public Double indexStateTtl = 1.5D;
+
+ @Parameter(names = {"--index-global-enabled"}, description = "Whether to
update index for the old partition path "
+ + "if same key record with different partition path came in, default
false")
+ public Boolean indexGlobalEnabled = false;
+
+ @Parameter(names = {"--index-partition-regex"},
+ description = "Whether to load partitions in state if partition path
matching， default *")
+ public String indexPartitionRegex = ".*";
+
+ @Parameter(names = {"--avro-schema-path"}, description = "Avro schema file
path, the parsed schema is used for deserialization")
+ public String avroSchemaPath = "";
+
+ @Parameter(names = {"--avro-schema"}, description = "Avro schema string, the
parsed schema is used for deserialization")
+ public String avroSchema = "";
+
+ @Parameter(names = {"--utc-timezone"}, description = "Use UTC timezone or
local timezone to the conversion between epoch"
+ + " time and LocalDateTime. Hive 0.x/1.x/2.x use local timezone. But
Hive 3.x"
+ + " use UTC timezone, by default true")
+ public Boolean utcTimezone = true;
+
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
hudi-bot edited a comment on pull request #3134: URL: https://github.com/apache/hudi/pull/3134#issuecomment-865885733 ## CI report: * 92bbb23756ffa63d7dd77a8a9285f3dd7e94d55c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=423) * cc2e75755d4c20cd7c53c84d6d8e12464e45a073 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 15cb914ed60ef660fdd4690eb80d2c6bd880f199 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=436) * 90781844a82336b56dbf32a39cf9d65874c5a57e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1884) MergeInto Support Partial Update For COW
[
https://issues.apache.org/jira/browse/HUDI-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pengzhiwei updated HUDI-1884:
-
Summary: MergeInto Support Partial Update For COW (was: Support Partial
Update For MergeInto)
> MergeInto Support Partial Update For COW
> -
>
> Key: HUDI-1884
> URL: https://issues.apache.org/jira/browse/HUDI-1884
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
>
> Support partial update for merge-into statement, just like this:
> {code:java}
> merge into h0
> using s0
> on s0.id = h0.id
> when matched then update set price = s0.price + 10
> when not matched then insert *{code}
> Allow to update partial fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] danny0405 commented on a change in pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
danny0405 commented on a change in pull request #3134:
URL: https://github.com/apache/hudi/pull/3134#discussion_r658615870
##
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapFunction.java
##
@@ -168,32 +175,69 @@ private HoodieFlinkTable getTable() {
* @param partitionPath The partition path
*/
@SuppressWarnings("unchecked")
- private void loadRecords(String partitionPath, Collector out) {
+ private void loadRecords(String partitionPath, Collector out) throws
Exception {
long start = System.currentTimeMillis();
+
BaseFileUtils fileUtils =
BaseFileUtils.getInstance(this.hoodieTable.getBaseFileFormat());
-List latestBaseFiles =
-HoodieIndexUtils.getLatestBaseFilesForPartition(partitionPath,
this.hoodieTable);
-LOG.info("All baseFile in partition {} size = {}", partitionPath,
latestBaseFiles.size());
+Schema schema = new
TableSchemaResolver(this.hoodieTable.getMetaClient()).getTableAvroSchema();
final int parallelism = getRuntimeContext().getNumberOfParallelSubtasks();
final int maxParallelism =
getRuntimeContext().getMaxNumberOfParallelSubtasks();
final int taskID = getRuntimeContext().getIndexOfThisSubtask();
-for (HoodieBaseFile baseFile : latestBaseFiles) {
- boolean shouldLoad = KeyGroupRangeAssignment.assignKeyToParallelOperator(
- baseFile.getFileId(), maxParallelism, parallelism) == taskID;
- if (shouldLoad) {
-LOG.info("Load records from file {}.", baseFile);
-final List hoodieKeys;
+Option latestCommitTime =
this.hoodieTable.getMetaClient().getCommitsTimeline()
+.filterCompletedInstants().lastInstant();
+
+if (latestCommitTime.isPresent()) {
+ List fileSlices = this.hoodieTable.getSliceView()
+ .getLatestFileSlicesBeforeOrOn(partitionPath,
latestCommitTime.get().getTimestamp(), true)
+ .collect(toList());
+
+ for (FileSlice fileSlice : fileSlices) {
+if (!shouldLoadFileId(fileSlice.getFileId(), maxParallelism,
parallelism, taskID)) {
+ continue;
+}
+LOG.info("Load records from {}.", fileSlice);
+
+// load parquet records
+fileSlice.getBaseFile().ifPresent(baseFile -> {
+ // filter out crushed files
+ if (baseFile.getFileSize() <= 0) {
+return;
+ }
+
+ final List hoodieKeys;
+ try {
+hoodieKeys =
+fileUtils.fetchRecordKeyPartitionPath(this.hadoopConf, new
Path(baseFile.getPath()));
+ } catch (Exception e) {
+throw new HoodieException(String.format("Error when loading record
keys from file: %s", baseFile), e);
+ }
+
+ for (HoodieKey hoodieKey : hoodieKeys) {
+out.collect((O) new IndexRecord(generateHoodieRecord(hoodieKey,
fileSlice)));
+ }
+});
+
+// load avro log records
+List logPaths = fileSlice.getLogFiles()
+// filter out crushed files
+.filter(logFile -> logFile.getFileSize() > 0)
+.map(logFile -> logFile.getPath().toString())
+.collect(toList());
+HoodieMergedLogRecordScanner scanner = scanLog(logPaths, schema,
latestCommitTime.get().getTimestamp());
+
+final List hoodieKeys = new ArrayList<>();
try {
- hoodieKeys =
- fileUtils.fetchRecordKeyPartitionPath(this.hadoopConf, new
Path(baseFile.getPath()));
+ for (String recordKey : scanner.getRecords().keySet()) {
+hoodieKeys.add(new HoodieKey(recordKey, partitionPath));
Review comment:
Can we send the keys directly and do not buffer the records ? The cache
takes too much memory footprint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 15cb914ed60ef660fdd4690eb80d2c6bd880f199 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=436) * 90781844a82336b56dbf32a39cf9d65874c5a57e UNKNOWN * bf4d7a1ca7023387301b812d40813051f7b4f720 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
hudi-bot edited a comment on pull request #3134: URL: https://github.com/apache/hudi/pull/3134#issuecomment-865885733 ## CI report: * 92bbb23756ffa63d7dd77a8a9285f3dd7e94d55c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=423) * cc2e75755d4c20cd7c53c84d6d8e12464e45a073 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=437) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 opened a new pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
pengzhiwei2018 opened a new pull request #3154: URL: https://github.com/apache/hudi/pull/3154 ## What is the purpose of the pull request MergeInto Support Partial Update For COW. ## Brief change log - Complete the missing UPDATE ACTION fields with the target table fields. e.g. > merge into h0 using s0 on h0.id = s0.id when matched then update set price = s0.price, ts = s0.ts; we complete the missing fields to the update action, like this: > merge into h0 using s0 on h0.id = s0.id when matched then update set id = h0.id, name = h0.name, price = s0.price, ts = s0.ts; ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1884) MergeInto Support Partial Update For COW
[
https://issues.apache.org/jira/browse/HUDI-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-1884:
-
Labels: pull-request-available (was: )
> MergeInto Support Partial Update For COW
> -
>
> Key: HUDI-1884
> URL: https://issues.apache.org/jira/browse/HUDI-1884
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
>
> Support partial update for merge-into statement, just like this:
> {code:java}
> merge into h0
> using s0
> on s0.id = h0.id
> when matched then update set price = s0.price + 10
> when not matched then insert *{code}
> Allow to update partial fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] hudi-bot commented on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
hudi-bot commented on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868360998 ## CI report: * f0f27c22c58a6e95e30f9195785a05321c70cbe1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * e1d659e1b04a7b262d7ee65bc5706e2d355ca5a3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=433) * cf8dac80c88f03cd2efb788f454aea5dd9904246 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
hudi-bot edited a comment on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868360998 ## CI report: * f0f27c22c58a6e95e30f9195785a05321c70cbe1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=438) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 15cb914ed60ef660fdd4690eb80d2c6bd880f199 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=436) * 90781844a82336b56dbf32a39cf9d65874c5a57e UNKNOWN * bf4d7a1ca7023387301b812d40813051f7b4f720 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=439) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * e1d659e1b04a7b262d7ee65bc5706e2d355ca5a3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=433) * cf8dac80c88f03cd2efb788f454aea5dd9904246 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=440) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
hudi-bot edited a comment on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868360998 ## CI report: * f0f27c22c58a6e95e30f9195785a05321c70cbe1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=438) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
hudi-bot edited a comment on pull request #3134: URL: https://github.com/apache/hudi/pull/3134#issuecomment-865885733 ## CI report: * cc2e75755d4c20cd7c53c84d6d8e12464e45a073 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=437) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a change in pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
yuzhaojing commented on a change in pull request #3134:
URL: https://github.com/apache/hudi/pull/3134#discussion_r658658537
##
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapFunction.java
##
@@ -168,32 +175,69 @@ private HoodieFlinkTable getTable() {
* @param partitionPath The partition path
*/
@SuppressWarnings("unchecked")
- private void loadRecords(String partitionPath, Collector out) {
+ private void loadRecords(String partitionPath, Collector out) throws
Exception {
long start = System.currentTimeMillis();
+
BaseFileUtils fileUtils =
BaseFileUtils.getInstance(this.hoodieTable.getBaseFileFormat());
-List latestBaseFiles =
-HoodieIndexUtils.getLatestBaseFilesForPartition(partitionPath,
this.hoodieTable);
-LOG.info("All baseFile in partition {} size = {}", partitionPath,
latestBaseFiles.size());
+Schema schema = new
TableSchemaResolver(this.hoodieTable.getMetaClient()).getTableAvroSchema();
final int parallelism = getRuntimeContext().getNumberOfParallelSubtasks();
final int maxParallelism =
getRuntimeContext().getMaxNumberOfParallelSubtasks();
final int taskID = getRuntimeContext().getIndexOfThisSubtask();
-for (HoodieBaseFile baseFile : latestBaseFiles) {
- boolean shouldLoad = KeyGroupRangeAssignment.assignKeyToParallelOperator(
- baseFile.getFileId(), maxParallelism, parallelism) == taskID;
- if (shouldLoad) {
-LOG.info("Load records from file {}.", baseFile);
-final List hoodieKeys;
+Option latestCommitTime =
this.hoodieTable.getMetaClient().getCommitsTimeline()
+.filterCompletedInstants().lastInstant();
+
+if (latestCommitTime.isPresent()) {
+ List fileSlices = this.hoodieTable.getSliceView()
+ .getLatestFileSlicesBeforeOrOn(partitionPath,
latestCommitTime.get().getTimestamp(), true)
+ .collect(toList());
+
+ for (FileSlice fileSlice : fileSlices) {
+if (!shouldLoadFileId(fileSlice.getFileId(), maxParallelism,
parallelism, taskID)) {
+ continue;
+}
+LOG.info("Load records from {}.", fileSlice);
+
+// load parquet records
+fileSlice.getBaseFile().ifPresent(baseFile -> {
+ // filter out crushed files
+ if (baseFile.getFileSize() <= 0) {
+return;
+ }
+
+ final List hoodieKeys;
+ try {
+hoodieKeys =
+fileUtils.fetchRecordKeyPartitionPath(this.hadoopConf, new
Path(baseFile.getPath()));
+ } catch (Exception e) {
+throw new HoodieException(String.format("Error when loading record
keys from file: %s", baseFile), e);
+ }
+
+ for (HoodieKey hoodieKey : hoodieKeys) {
+out.collect((O) new IndexRecord(generateHoodieRecord(hoodieKey,
fileSlice)));
+ }
+});
+
+// load avro log records
+List logPaths = fileSlice.getLogFiles()
+// filter out crushed files
+.filter(logFile -> logFile.getFileSize() > 0)
+.map(logFile -> logFile.getPath().toString())
+.collect(toList());
+HoodieMergedLogRecordScanner scanner = scanLog(logPaths, schema,
latestCommitTime.get().getTimestamp());
+
+final List hoodieKeys = new ArrayList<>();
try {
- hoodieKeys =
- fileUtils.fetchRecordKeyPartitionPath(this.hadoopConf, new
Path(baseFile.getPath()));
+ for (String recordKey : scanner.getRecords().keySet()) {
+hoodieKeys.add(new HoodieKey(recordKey, partitionPath));
Review comment:
yes, this cache use memory unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
hudi-bot edited a comment on pull request #3134: URL: https://github.com/apache/hudi/pull/3134#issuecomment-865885733 ## CI report: * cc2e75755d4c20cd7c53c84d6d8e12464e45a073 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=437) * b91eda365c478318b1cec6b6bf0b4c98ecd07a23 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
hudi-bot edited a comment on pull request #3134: URL: https://github.com/apache/hudi/pull/3134#issuecomment-865885733 ## CI report: * cc2e75755d4c20cd7c53c84d6d8e12464e45a073 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=437) * b91eda365c478318b1cec6b6bf0b4c98ecd07a23 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=441) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 90781844a82336b56dbf32a39cf9d65874c5a57e UNKNOWN * bf4d7a1ca7023387301b812d40813051f7b4f720 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=439) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * cf8dac80c88f03cd2efb788f454aea5dd9904246 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=440) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
hudi-bot edited a comment on pull request #3134: URL: https://github.com/apache/hudi/pull/3134#issuecomment-865885733 ## CI report: * b91eda365c478318b1cec6b6bf0b4c98ecd07a23 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=441) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r658703745
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##
@@ -297,7 +295,37 @@ public String getTopicName() {
return topicName;
}
- public HashMap getKafkaParams() {
+ public Map getKafkaParams() {
+return kafkaParams;
+ }
+
+ private static Map excludeHoodieConfigs(TypedProperties
props) {
Review comment:
@leesf done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
hudi-bot edited a comment on pull request #3092: URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673 ## CI report: * 1ae031303ebfa92fd153b8b6c0a404d1aac39a0a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=393) * e90cbcdc4a4926230d2440b486c52e39850cdb1a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
codecov-commenter edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868296609 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3151](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf4d7a1) into [master](https://codecov.io/gh/apache/hudi/commit/01ad449ad68d9c77ee1493fcdf833df53df6106a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01ad449) will **decrease** coverage by `49.11%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3151 +/- ## - Coverage 52.19% 3.07% -49.12% + Complexity 2659 82 -2577 Files 335 279 -56 Lines 14981 10855 -4126 Branches 15051107 -398 - Hits 7819 334 -7485 - Misses 6536 10495 +3959 + Partials626 26 -600 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (∅)` | | | hudicommon | `?` | | | hudihadoopmr | `?` | | | hudisync | `5.38% <ø> (?)` | | | huditimelineservice | `?` | | | hudiutilities | `9.52% <ø> (-60.13%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <ø> (ø)` | | | [...a/org/apache/hudi/client/AbstractHoodieClient.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZUNsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | | | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/client/AsyncCleanerService.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Bc3luY0NsZWFuZXJTZXJ2aWNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../org/apache/hudi/client/CompactionAdminClient.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Db21wYWN0aW9uQWRtaW5DbGllbnQuamF2YQ==) | `0.00% <ø> (ø)` | | | [.../org/apache/hudi/client/ReplaceArchivalHelper.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referr
[GitHub] [hudi] codecov-commenter commented on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
codecov-commenter commented on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868445474 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3154](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f0f27c2) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `43.16%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3154 +/- ## - Coverage 46.24% 3.07% -43.17% + Complexity 5362 82 -5280 Files 920 279 -641 Lines 39842 10855-28987 Branches 42911107 -3184 - Hits 18425 334-18091 + Misses19535 10495 -9040 + Partials 1882 26 -1856 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.52% <ø> (-48.77%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
hudi-bot edited a comment on pull request #3092: URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673 ## CI report: * e90cbcdc4a4926230d2440b486c52e39850cdb1a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
hudi-bot edited a comment on pull request #3092: URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673 ## CI report: * e90cbcdc4a4926230d2440b486c52e39850cdb1a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442) * 2d5c2588f9088daef2800600558ebaa7d6d05212 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] veenaypatil commented on a change in pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
veenaypatil commented on a change in pull request #3092:
URL: https://github.com/apache/hudi/pull/3092#discussion_r658715036
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##
@@ -473,7 +476,9 @@ public void refreshTimeline() throws IOException {
boolean success = writeClient.commit(instantTime, writeStatusRDD,
Option.of(checkpointCommitMetadata));
if (success) {
LOG.info("Commit " + instantTime + " successful!");
-
+if (this.props.getBoolean(ENABLE_KAFKA_COMMIT_OFFSET,
DEFAULT_ENABLE_KAFKA_COMMIT_OFFSET)) {
Review comment:
@n3nash @vinothchandar added `onCommit` method to Source, let me know
if this approach works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
codecov-commenter edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868296609 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3151](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf4d7a1) into [master](https://codecov.io/gh/apache/hudi/commit/01ad449ad68d9c77ee1493fcdf833df53df6106a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01ad449) will **decrease** coverage by `36.01%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3151 +/- ## = - Coverage 52.19% 16.17% -36.02% + Complexity 2659 466 -2193 = Files 335 279 -56 Lines 1498110855 -4126 Branches 1505 1107 -398 = - Hits 7819 1756 -6063 - Misses 6536 8939 +2403 + Partials626 160 -466 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (∅)` | | | hudicommon | `?` | | | hudihadoopmr | `?` | | | hudisync | `5.38% <ø> (?)` | | | huditimelineservice | `?` | | | hudiutilities | `58.25% <ø> (-11.40%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ava/org/apache/hudi/async/AsyncCompactService.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `0.00% <ø> (ø)` | | | [...a/org/apache/hudi/client/AbstractHoodieClient.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZUNsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | | | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/client/AsyncCleanerService.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Bc3luY0NsZWFuZXJTZXJ2aWNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../org/apache/hudi/client/CompactionAdminClient.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Db21wYWN0aW9uQWRtaW5DbGllbnQuamF2YQ==) | `0.00% <ø> (ø)` | | | [.../org/apache/hudi/client/ReplaceArchivalHelper.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
codecov-commenter edited a comment on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868445474 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3154](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f0f27c2) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `30.06%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3154 +/- ## = - Coverage 46.24% 16.17% -30.07% + Complexity 5362 466 -4896 = Files 920 279 -641 Lines 3984210855-28987 Branches 4291 1107 -3184 = - Hits 18425 1756-16669 + Misses19535 8939-10596 + Partials 1882 160 -1722 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.25% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 90781844a82336b56dbf32a39cf9d65874c5a57e UNKNOWN * bf4d7a1ca7023387301b812d40813051f7b4f720 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=439) * 2b260644461696901d85314402c00813a5b20dfd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
codecov-commenter edited a comment on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868445474 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3154](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f0f27c2) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `17.84%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3154 +/- ## = - Coverage 46.24% 28.40% -17.85% + Complexity 5362 1254 -4108 = Files 920 375 -545 Lines 3984214306-25536 Branches 4291 1459 -2832 = - Hits 18425 4063-14362 + Misses19535 9946 -9589 + Partials 1882 297 -1585 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `22.29% <ø> (-8.17%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `58.25% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3154/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
codecov-commenter edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868296609 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3151](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf4d7a1) into [master](https://codecov.io/gh/apache/hudi/commit/01ad449ad68d9c77ee1493fcdf833df53df6106a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01ad449) will **decrease** coverage by `12.52%`. > The diff coverage is `15.76%`. > :exclamation: Current head bf4d7a1 differs from pull request most recent head 2b26064. Consider uploading reports for the commit 2b26064 to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3151 +/- ## = - Coverage 52.19% 39.67% -12.53% - Complexity 2659 3948 +1289 = Files 335 746 +411 Lines 1498132137+17156 Branches 1505 3269 +1764 = + Hits 781912749 +4930 - Misses 653618129+11593 - Partials626 1259 +633 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <20.83%> (+1.12%)` | :arrow_up: | | hudiclient | `16.45% <15.03%> (∅)` | | | hudicommon | `47.58% <ø> (-7.16%)` | :arrow_down: | | hudiflink | `59.61% <ø> (?)` | | | hudihadoopmr | `51.29% <ø> (+17.99%)` | :arrow_up: | | hudisync | `5.38% <ø> (?)` | | | huditimelineservice | `?` | | | hudiutilities | `58.25% <ø> (-11.40%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...a/org/apache/hudi/cli/HoodieTableHeaderFields.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL0hvb2RpZVRhYmxlSGVhZGVyRmllbGRzLmphdmE=) | `0.00% <ø> (ø)` | | | [...rg/apache/hudi/cli/commands/SavepointsCommand.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1NhdmVwb2ludHNDb21tYW5kLmphdmE=) | `13.84% <0.00%> (-0.44%)` | :arrow_down: | | [...org/apache/hudi/cli/utils/InputStreamConsumer.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL3V0aWxzL0lucHV0U3RyZWFtQ29uc3VtZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/scala/org/apache/hudi/cli/DedupeSparkJob.scala](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL2NsaS9EZWR1cGVTcGFya0pvYi5zY2FsYQ==) | `0.00% <0.00%> (ø)` | | | [.../main/scala/org/apache/hudi/cli/SparkHelpers.scala](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL2NsaS9TcGFya0hlbHBlcn
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
codecov-commenter edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868296609 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3151](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bf4d7a1) into [master](https://codecov.io/gh/apache/hudi/commit/01ad449ad68d9c77ee1493fcdf833df53df6106a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01ad449) will **decrease** coverage by `8.02%`. > The diff coverage is `15.76%`. > :exclamation: Current head bf4d7a1 differs from pull request most recent head 2b26064. Consider uploading reports for the commit 2b26064 to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3151 +/- ## - Coverage 52.19% 44.16% -8.03% - Complexity 2659 4573+1914 Files 335 824 +489 Lines 1498136499 +21518 Branches 1505 3939+2434 + Hits 781916119+8300 - Misses 653618635 +12099 - Partials626 1745+1119 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.95% <20.83%> (+1.12%)` | :arrow_up: | | hudiclient | `16.45% <15.03%> (∅)` | | | hudicommon | `47.58% <ø> (-7.16%)` | :arrow_down: | | hudiflink | `59.61% <ø> (?)` | | | hudihadoopmr | `51.29% <ø> (+17.99%)` | :arrow_up: | | hudisparkdatasource | `67.06% <ø> (?)` | | | hudisync | `54.05% <ø> (?)` | | | huditimelineservice | `64.36% <ø> (-0.94%)` | :arrow_down: | | hudiutilities | `58.25% <ø> (-11.40%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3151?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...a/org/apache/hudi/cli/HoodieTableHeaderFields.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL0hvb2RpZVRhYmxlSGVhZGVyRmllbGRzLmphdmE=) | `0.00% <ø> (ø)` | | | [...rg/apache/hudi/cli/commands/SavepointsCommand.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1NhdmVwb2ludHNDb21tYW5kLmphdmE=) | `13.84% <0.00%> (-0.44%)` | :arrow_down: | | [...org/apache/hudi/cli/utils/InputStreamConsumer.java](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL3V0aWxzL0lucHV0U3RyZWFtQ29uc3VtZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/scala/org/apache/hudi/cli/DedupeSparkJob.scala](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGkvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL2NsaS9EZWR1cGVTcGFya0pvYi5zY2FsYQ==) | `0.00% <0.00%> (ø)` | | | [.../main/scala/org/apache/hudi/cli/SparkHelpers.scala](https://codecov.io/gh/apache/hudi/pull/3151/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1j
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
codecov-commenter edited a comment on pull request #3154: URL: https://github.com/apache/hudi/pull/3154#issuecomment-868445474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
hudi-bot edited a comment on pull request #3092: URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673 ## CI report: * e90cbcdc4a4926230d2440b486c52e39850cdb1a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=442) * 2d5c2588f9088daef2800600558ebaa7d6d05212 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 90781844a82336b56dbf32a39cf9d65874c5a57e UNKNOWN * bf4d7a1ca7023387301b812d40813051f7b4f720 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=439) * 2b260644461696901d85314402c00813a5b20dfd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=444) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan opened a new pull request #3155: [Do-No-Merge][WIP] Running TestCleaner tests repeatedly
nsivabalan opened a new pull request #3155: URL: https://github.com/apache/hudi/pull/3155 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3155: [Do-No-Merge][WIP] Running TestCleaner tests repeatedly
hudi-bot commented on pull request #3155: URL: https://github.com/apache/hudi/pull/3155#issuecomment-868495040 ## CI report: * 9ce57d1973d9e041ac993a6b42337546f4f845af UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3155: [Do-No-Merge][WIP] Running TestCleaner tests repeatedly
hudi-bot edited a comment on pull request #3155: URL: https://github.com/apache/hudi/pull/3155#issuecomment-868495040 ## CI report: * 9ce57d1973d9e041ac993a6b42337546f4f845af Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=445) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
codecov-commenter edited a comment on pull request #3134: URL: https://github.com/apache/hudi/pull/3134#issuecomment-866110119 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3134?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3134](https://codecov.io/gh/apache/hudi/pull/3134?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b91eda3) into [master](https://codecov.io/gh/apache/hudi/commit/7bd517a82f9da3a6b42f6786da1d8a23151bdd94?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7bd517a) will **increase** coverage by `3.67%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3134?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3134 +/- ## + Coverage 46.08% 49.76% +3.67% + Complexity 5319 393-4926 Files 914 66 -848 Lines 39596 2918 -36678 Branches 4269 319-3950 - Hits 18249 1452 -16797 + Misses19480 1329 -18151 + Partials 1867 137-1730 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `?` | | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `49.76% <ø> (-8.66%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3134?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.63%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...org/apache/hudi/utilities/HDFSParquetImporter.java](https://codecov.io/gh/apache/hudi/pull/3134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hERlNQYXJxdWV0SW1wb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-71.82%)` | :arrow_down: | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/3134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | :arrow_down: | | [...g/apache/hudi/utilities/schema/SchemaPro
[GitHub] [hudi] fanaticjo commented on a change in pull request #3035: [HUDI-1936] Introduce a optional property for conditional upsert
fanaticjo commented on a change in pull request #3035:
URL: https://github.com/apache/hudi/pull/3035#discussion_r658767757
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithCustomAvroPayload.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.ColumnNotFoundException;
+import org.apache.hudi.exception.UpdateKeyNotFoundException;
+import org.apache.hudi.exception.WriteOperationException;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+import java.util.stream.Collectors;
+
+/**
+ * subclass of OverwriteWithLatestAvroPayload used for delta streamer.
+ *
+ *
+ * combineAndGetUpdateValue - Accepts the column names to be updated;
+ * splitKeys - Split keys based upon keys;
+ *
+ */
+public class OverwriteWithCustomAvroPayload extends
OverwriteWithLatestAvroPayload {
+
+ public OverwriteWithCustomAvroPayload(GenericRecord record, Comparable
orderingVal) {
+super(record, orderingVal);
+ }
+
+ /**
+ * split keys over.
+ */
+ public List splitKeys(String keys) throws UpdateKeyNotFoundException
{
+if (keys == null) {
+ throw new UpdateKeyNotFoundException("keys cannot be null");
+} else if (keys.equals("")) {
+ throw new UpdateKeyNotFoundException("keys cannot be blank");
+} else {
+ return Arrays.stream(keys.split(",")).collect(Collectors.toList());
+}
+ }
+
+ /**
+ * check column exi.
+ */
+ public boolean checkColumnExists(List keys, Schema schema) {
+List field = schema.getFields();
+List common = new ArrayList<>();
+for (Schema.Field columns : field) {
+ if (keys.contains(columns.name())) {
+common.add(columns);
+ }
+}
+return common.size() == keys.size();
+ }
+
+ @Override
+ public Option combineAndGetUpdateValue(IndexedRecord
currentValue, Schema schema, Properties properties)
+ throws WriteOperationException, IOException, ColumnNotFoundException,
UpdateKeyNotFoundException {
+
+if
(!properties.getProperty("hoodie.datasource.write.operation").equals("upsert"))
{
+ throw new WriteOperationException("write should be upsert");
+}
+
+Option recordOption = getInsertValue(schema);
+
+if (!recordOption.isPresent()) {
+ return Option.empty();
+}
+
Review comment:
yes user can set different values for different batches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] fanaticjo commented on a change in pull request #3035: [HUDI-1936] Introduce a optional property for conditional upsert
fanaticjo commented on a change in pull request #3035:
URL: https://github.com/apache/hudi/pull/3035#discussion_r658767757
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithCustomAvroPayload.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.ColumnNotFoundException;
+import org.apache.hudi.exception.UpdateKeyNotFoundException;
+import org.apache.hudi.exception.WriteOperationException;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+import java.util.stream.Collectors;
+
+/**
+ * subclass of OverwriteWithLatestAvroPayload used for delta streamer.
+ *
+ *
+ * combineAndGetUpdateValue - Accepts the column names to be updated;
+ * splitKeys - Split keys based upon keys;
+ *
+ */
+public class OverwriteWithCustomAvroPayload extends
OverwriteWithLatestAvroPayload {
+
+ public OverwriteWithCustomAvroPayload(GenericRecord record, Comparable
orderingVal) {
+super(record, orderingVal);
+ }
+
+ /**
+ * split keys over.
+ */
+ public List splitKeys(String keys) throws UpdateKeyNotFoundException
{
+if (keys == null) {
+ throw new UpdateKeyNotFoundException("keys cannot be null");
+} else if (keys.equals("")) {
+ throw new UpdateKeyNotFoundException("keys cannot be blank");
+} else {
+ return Arrays.stream(keys.split(",")).collect(Collectors.toList());
+}
+ }
+
+ /**
+ * check column exi.
+ */
+ public boolean checkColumnExists(List keys, Schema schema) {
+List field = schema.getFields();
+List common = new ArrayList<>();
+for (Schema.Field columns : field) {
+ if (keys.contains(columns.name())) {
+common.add(columns);
+ }
+}
+return common.size() == keys.size();
+ }
+
+ @Override
+ public Option combineAndGetUpdateValue(IndexedRecord
currentValue, Schema schema, Properties properties)
+ throws WriteOperationException, IOException, ColumnNotFoundException,
UpdateKeyNotFoundException {
+
+if
(!properties.getProperty("hoodie.datasource.write.operation").equals("upsert"))
{
+ throw new WriteOperationException("write should be upsert");
+}
+
+Option recordOption = getInsertValue(schema);
+
+if (!recordOption.isPresent()) {
+ return Option.empty();
+}
+
Review comment:
yes user can set different values for different batches for cow it
working , mor will test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
hudi-bot edited a comment on pull request #3092: URL: https://github.com/apache/hudi/pull/3092#issuecomment-862136673 ## CI report: * 2d5c2588f9088daef2800600558ebaa7d6d05212 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=443) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3151: [HUDI-2067] Sync FlinkOptions config to FlinkStreamerConfig
hudi-bot edited a comment on pull request #3151: URL: https://github.com/apache/hudi/pull/3151#issuecomment-868252016 ## CI report: * 90781844a82336b56dbf32a39cf9d65874c5a57e UNKNOWN * 2b260644461696901d85314402c00813a5b20dfd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=444) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * cf8dac80c88f03cd2efb788f454aea5dd9904246 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=440) * a4596e734fbfc0701b8aa8e13b769994d776a425 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3134: [HUDI-2052] Support load logFile in BootstrapFunction
codecov-commenter edited a comment on pull request #3134: URL: https://github.com/apache/hudi/pull/3134#issuecomment-866110119 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3134?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3134](https://codecov.io/gh/apache/hudi/pull/3134?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b91eda3) into [master](https://codecov.io/gh/apache/hudi/commit/7bd517a82f9da3a6b42f6786da1d8a23151bdd94?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7bd517a) will **increase** coverage by `12.93%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3134?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3134 +/- ## = + Coverage 46.08% 59.02% +12.93% + Complexity 5319 1181 -4138 = Files 914 162 -752 Lines 39596 6369-33227 Branches 4269 671 -3598 = - Hits 18249 3759-14490 + Misses19480 2336-17144 + Partials 1867 274 -1593 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `66.85% <ø> (+36.40%)` | :arrow_up: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `49.76% <ø> (-8.66%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3134?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.63%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...org/apache/hudi/utilities/HDFSParquetImporter.java](https://codecov.io/gh/apache/hudi/pull/3134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hERlNQYXJxdWV0SW1wb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-71.82%)` | :arrow_down: | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/3134/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | :arrow_down: | |
[GitHub] [hudi] nsivabalan opened a new pull request #3156: [Do-Not-Merge][WIP] Repeated test runs to debug flaky tests. TestHoodieMergeOnReadTable#testRollbackWithDeltaAndCompactionCommitUsingFileLis
nsivabalan opened a new pull request #3156: URL: https://github.com/apache/hudi/pull/3156 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3156: [Do-Not-Merge][WIP] Repeated test runs to debug flaky tests. TestHoodieMergeOnReadTable#testRollbackWithDeltaAndCompactionCommitUsingFileList
hudi-bot commented on pull request #3156: URL: https://github.com/apache/hudi/pull/3156#issuecomment-868518502 ## CI report: * 253ab8e2d53c51208d15b26e740ca292366693d7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * cf8dac80c88f03cd2efb788f454aea5dd9904246 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=440) * a4596e734fbfc0701b8aa8e13b769994d776a425 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=446) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3156: [Do-Not-Merge][WIP] Repeated test runs to debug flaky tests. TestHoodieMergeOnReadTable#testRollbackWithDeltaAndCompactionCommitUsingFi
hudi-bot edited a comment on pull request #3156: URL: https://github.com/apache/hudi/pull/3156#issuecomment-868518502 ## CI report: * 253ab8e2d53c51208d15b26e740ca292366693d7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=447) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3155: [Do-No-Merge][WIP] Running TestCleaner tests repeatedly
hudi-bot edited a comment on pull request #3155: URL: https://github.com/apache/hudi/pull/3155#issuecomment-868495040 ## CI report: * 9ce57d1973d9e041ac993a6b42337546f4f845af Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=445) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #3153: [HUDI-2072] Add pre-commit validator framework
bvaradar commented on pull request #3153: URL: https://github.com/apache/hudi/pull/3153#issuecomment-868548098 Thanks a lot @satishkotha for the quick turnaround. Will look into this over the weekend -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HUDI-2061) Incorrect Schema Inference For Schema Evolved Table
[ https://issues.apache.org/jira/browse/HUDI-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengzhiwei resolved HUDI-2061. -- Resolution: Fixed > Incorrect Schema Inference For Schema Evolved Table > --- > > Key: HUDI-2061 > URL: https://issues.apache.org/jira/browse/HUDI-2061 > Project: Apache Hudi > Issue Type: Bug > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > The test case *HoodieSparkSqlWriterSuite(test schema evolution for xx)* often > crash because currently we have not specify the right schema to the > *DataSource.apply* in DefaultSource#getBaseFileOnlyView. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3120: [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And Del…
hudi-bot edited a comment on pull request #3120: URL: https://github.com/apache/hudi/pull/3120#issuecomment-864760893 ## CI report: * aaca30fffd1ea37f803f51ef3cf49c59ed79badc UNKNOWN * fcd06c8bccfc90b272b51d3511094e6617ec25bd UNKNOWN * 96947d0419df5f8bab10072eb64afecd29326e55 UNKNOWN * a4596e734fbfc0701b8aa8e13b769994d776a425 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=446) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3156: [Do-Not-Merge][WIP] Repeated test runs to debug flaky tests. TestHoodieMergeOnReadTable#testRollbackWithDeltaAndCompactionCommitUsingFi
hudi-bot edited a comment on pull request #3156: URL: https://github.com/apache/hudi/pull/3156#issuecomment-868518502 ## CI report: * 253ab8e2d53c51208d15b26e740ca292366693d7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=447) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-2073) The sparkJob of hoodieClusteringJob running through sparkSubmit will not quit even it is finished or failed.
Yue Zhang created HUDI-2073: --- Summary: The sparkJob of hoodieClusteringJob running through sparkSubmit will not quit even it is finished or failed. Key: HUDI-2073 URL: https://issues.apache.org/jira/browse/HUDI-2073 Project: Apache Hudi Issue Type: Bug Reporter: Yue Zhang Users can launch HoodieClusteringJob through sparkSubmit to # Scheduling clustering # Execute clustering But these spark jobs will never finished and SparkSubmit never quit even jobs are finished or failed. This is because clustering job will init a SparkRDDWriteClient to doSchedule or do cluster But did not close this client after that. It will cause that `jsc.stop();` can't kill this sparkJob and hang forever. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3092: [HUDI-1910] Commit Offset to Kafka after successful Hudi commit
codecov-commenter edited a comment on pull request #3092: URL: https://github.com/apache/hudi/pull/3092#issuecomment-864517429 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3092](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2d5c258) into [master](https://codecov.io/gh/apache/hudi/commit/23dbc09a0d5d677ed3728dbd5357320590ca2833?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (23dbc09) will **decrease** coverage by `43.17%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3092 +/- ## - Coverage 46.24% 3.06% -43.18% + Complexity 5362 82 -5280 Files 920 279 -641 Lines 39842 10884-28958 Branches 42911106 -3185 - Hits 18425 334-18091 + Misses19535 10524 -9011 + Partials 1882 26 -1856 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.43% <0.00%> (-48.87%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3092?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-71.19%)` | :arrow_down: | | [...i/utilities/deltastreamer/SourceFormatAdapter.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvU291cmNlRm9ybWF0QWRhcHRlci5qYXZh) | `0.00% <0.00%> (-86.49%)` | :arrow_down: | | [...g/apache/hudi/utilities/sources/AvroDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0RGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...apache/hudi/utilities/sources/AvroKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb0thZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3092/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...che/hudi/utilities/sources/HiveIncrPu
[GitHub] [hudi] zhangyue19921010 opened a new pull request #3157: [HUDI-2073] Fix the bug of hoodieClusteringJob never quit
zhangyue19921010 opened a new pull request #3157:
URL: https://github.com/apache/hudi/pull/3157
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a
pull request.*
## What is the purpose of the pull request
Users can launch HoodieClusteringJob through sparkSubmit to
1. Scheduling clustering
2. Execute clustering
But these spark jobs will never finished and SparkSubmit never quit even
jobs are finished or failed.
This is because clustering job will init a SparkRDDWriteClient to doSchedule
or do cluster But did not close this client after that. It will cause that
`jsc.stop();` can't kill this sparkJob and hang forever.
This PR is trying to fix this bug.
## Brief change log
add a try(client) {} to make sure that this client will be closed no
matter this job is successful or failed after used.
Also this pr is tested on my local env and works fine.
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test
coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please
describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HUDI-2073) The sparkJob of hoodieClusteringJob running through sparkSubmit will not quit even it is finished or failed.
[ https://issues.apache.org/jira/browse/HUDI-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2073: - Labels: pull-request-available (was: ) > The sparkJob of hoodieClusteringJob running through sparkSubmit will not quit > even it is finished or failed. > > > Key: HUDI-2073 > URL: https://issues.apache.org/jira/browse/HUDI-2073 > Project: Apache Hudi > Issue Type: Bug >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > Users can launch HoodieClusteringJob through sparkSubmit to > # Scheduling clustering > # Execute clustering > But these spark jobs will never finished and SparkSubmit never quit even jobs > are finished or failed. > This is because clustering job will init a SparkRDDWriteClient to doSchedule > or do cluster But did not close this client after that. It will cause that > `jsc.stop();` can't kill this sparkJob and hang forever. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2073) The sparkJob of hoodieClusteringJob running through sparkSubmit will not quit even it is finished or failed.
[ https://issues.apache.org/jira/browse/HUDI-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17369525#comment-17369525 ] Yue Zhang commented on HUDI-2073: - I just make a pr to fix this bug. linked here https://github.com/apache/hudi/pull/3157 > The sparkJob of hoodieClusteringJob running through sparkSubmit will not quit > even it is finished or failed. > > > Key: HUDI-2073 > URL: https://issues.apache.org/jira/browse/HUDI-2073 > Project: Apache Hudi > Issue Type: Bug >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > Users can launch HoodieClusteringJob through sparkSubmit to > # Scheduling clustering > # Execute clustering > But these spark jobs will never finished and SparkSubmit never quit even jobs > are finished or failed. > This is because clustering job will init a SparkRDDWriteClient to doSchedule > or do cluster But did not close this client after that. It will cause that > `jsc.stop();` can't kill this sparkJob and hang forever. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot commented on pull request #3157: [HUDI-2073] Fix the bug of hoodieClusteringJob never quit
hudi-bot commented on pull request #3157: URL: https://github.com/apache/hudi/pull/3157#issuecomment-868568023 ## CI report: * d685b7b110bc7441aed67c2ac5bbf726992f7ddb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3157: [HUDI-2073] Fix the bug of hoodieClusteringJob never quit
hudi-bot edited a comment on pull request #3157: URL: https://github.com/apache/hudi/pull/3157#issuecomment-868568023 ## CI report: * d685b7b110bc7441aed67c2ac5bbf726992f7ddb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=448) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] fanaticjo commented on a change in pull request #3035: [HUDI-1936] Introduce a optional property for conditional upsert
fanaticjo commented on a change in pull request #3035:
URL: https://github.com/apache/hudi/pull/3035#discussion_r658855139
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithCustomAvroPayload.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.ColumnNotFoundException;
+import org.apache.hudi.exception.UpdateKeyNotFoundException;
+import org.apache.hudi.exception.WriteOperationException;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+import java.util.stream.Collectors;
+
+/**
+ * subclass of OverwriteWithLatestAvroPayload used for delta streamer.
+ *
+ *
+ * combineAndGetUpdateValue - Accepts the column names to be updated;
+ * splitKeys - Split keys based upon keys;
+ *
+ */
+public class OverwriteWithCustomAvroPayload extends
OverwriteWithLatestAvroPayload {
+
+ public OverwriteWithCustomAvroPayload(GenericRecord record, Comparable
orderingVal) {
+super(record, orderingVal);
+ }
+
+ /**
+ * split keys over.
+ */
+ public List splitKeys(String keys) throws UpdateKeyNotFoundException
{
+if (keys == null) {
+ throw new UpdateKeyNotFoundException("keys cannot be null");
+} else if (keys.equals("")) {
+ throw new UpdateKeyNotFoundException("keys cannot be blank");
+} else {
+ return Arrays.stream(keys.split(",")).collect(Collectors.toList());
+}
+ }
+
+ /**
+ * check column exi.
+ */
+ public boolean checkColumnExists(List keys, Schema schema) {
+List field = schema.getFields();
+List common = new ArrayList<>();
+for (Schema.Field columns : field) {
+ if (keys.contains(columns.name())) {
+common.add(columns);
+ }
+}
+return common.size() == keys.size();
+ }
+
+ @Override
+ public Option combineAndGetUpdateValue(IndexedRecord
currentValue, Schema schema, Properties properties)
+ throws WriteOperationException, IOException, ColumnNotFoundException,
UpdateKeyNotFoundException {
+
+if
(!properties.getProperty("hoodie.datasource.write.operation").equals("upsert"))
{
+ throw new WriteOperationException("write should be upsert");
+}
+
+Option recordOption = getInsertValue(schema);
+
+if (!recordOption.isPresent()) {
+ return Option.empty();
+}
+
+GenericRecord existingRecord = (GenericRecord) currentValue;
+GenericRecord incomingRecord = (GenericRecord) recordOption.get();
+List keys =
splitKeys(properties.getProperty("hoodie.update.keys"));
Review comment:
i dont think DataSourceWriteOptions is accessible from model package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
