[GitHub] [hudi] codecov-commenter edited a comment on pull request #3288: [WIP] test 1771
codecov-commenter edited a comment on pull request #3288: URL: https://github.com/apache/hudi/pull/3288#issuecomment-881647912 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3288?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3288](https://codecov.io/gh/apache/hudi/pull/3288?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fc77605) into [master](https://codecov.io/gh/apache/hudi/commit/50c2b76d725a71608a38217370b1ac45cedae405?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (50c2b76) will **decrease** coverage by `31.75%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3288?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3288 +/- ## = - Coverage 47.78% 16.03% -31.76% + Complexity 5557 503 -5054 = Files 936 284 -652 Lines 4159611875-29721 Branches 4185 988 -3197 = - Hits 19877 1904-17973 + Misses19949 9807-10142 + Partials 1770 164 -1606 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.77% <ø> (+0.50%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3288?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUNyZWF0ZUhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/me
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382482#comment-17382482 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 01fa47d89991e0de17d8cbe50e6a8aae04ea7690 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=984) * de95dc113e89cf4ef8f03dd35ae89545ac19beee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=986) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 01fa47d89991e0de17d8cbe50e6a8aae04ea7690 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=984) * de95dc113e89cf4ef8f03dd35ae89545ac19beee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=986) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382481#comment-17382481 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 01fa47d89991e0de17d8cbe50e6a8aae04ea7690 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=984) * de95dc113e89cf4ef8f03dd35ae89545ac19beee UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 01fa47d89991e0de17d8cbe50e6a8aae04ea7690 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=984) * de95dc113e89cf4ef8f03dd35ae89545ac19beee UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3288: [WIP] test 1771
codecov-commenter edited a comment on pull request #3288: URL: https://github.com/apache/hudi/pull/3288#issuecomment-881647912 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3288?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3288](https://codecov.io/gh/apache/hudi/pull/3288?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fc77605) into [master](https://codecov.io/gh/apache/hudi/commit/50c2b76d725a71608a38217370b1ac45cedae405?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (50c2b76) will **decrease** coverage by `44.96%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3288?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3288 +/- ## - Coverage 47.78% 2.82% -44.97% + Complexity 5557 85 -5472 Files 936 284 -652 Lines 41596 11875-29721 Branches 4185 988 -3197 - Hits 19877 335-19542 + Misses19949 11514 -8435 + Partials 1770 26 -1744 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.99% <ø> (-50.27%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3288?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/hudi/pull/3288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUNyZWF0ZUhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3288/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://cod
[GitHub] [hudi] hudi-bot edited a comment on pull request #3288: [WIP] test 1771
hudi-bot edited a comment on pull request #3288: URL: https://github.com/apache/hudi/pull/3288#issuecomment-881594740 ## CI report: * 3d4aefc69186e21b521f9bd6b0d4fa7e0998df3a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=974) * fc7760579b982b3016244ad67e5361db53e42093 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=985) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3288: [WIP] test 1771
hudi-bot edited a comment on pull request #3288: URL: https://github.com/apache/hudi/pull/3288#issuecomment-881594740 ## CI report: * 3d4aefc69186e21b521f9bd6b0d4fa7e0998df3a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=974) * fc7760579b982b3016244ad67e5361db53e42093 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382478#comment-17382478 ] ASF GitHub Bot commented on HUDI-2161: -- hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 5b5c7c2fcdff2a5f5e0737da8c9d03da83c4a65c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=983) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 5b5c7c2fcdff2a5f5e0737da8c9d03da83c4a65c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=983) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382472#comment-17382472 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 01fa47d89991e0de17d8cbe50e6a8aae04ea7690 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=984) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 01fa47d89991e0de17d8cbe50e6a8aae04ea7690 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=984) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382469#comment-17382469 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 2cf38df103e1435c4239824f4908520727e38f69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=958) * 01fa47d89991e0de17d8cbe50e6a8aae04ea7690 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=984) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382468#comment-17382468 ] ASF GitHub Bot commented on HUDI-2161: -- codecov-commenter edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816086 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3247](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5b5c7c2) into [master](https://codecov.io/gh/apache/hudi/commit/50c2b76d725a71608a38217370b1ac45cedae405?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (50c2b76) will **decrease** coverage by `1.76%`. > The diff coverage is `62.06%`. [](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3247 +/- ## - Coverage 47.78% 46.01% -1.77% + Complexity 5557 4763 -794 Files 936 832 -104 Lines 4159638058-3538 Branches 4185 3809 -376 - Hits 1987717514-2363 + Misses1994918920-1029 + Partials 1770 1624 -146 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `23.00% <0.00%> (-11.53%)` | :arrow_down: | | hudicommon | `48.67% <26.66%> (-0.03%)` | :arrow_down: | | hudiflink | `59.36% <ø> (ø)` | | | hudihadoopmr | `52.02% <ø> (ø)` | | | hudisparkdatasource | `67.12% <73.52%> (-0.26%)` | :arrow_down: | | hudisync | `55.97% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/HoodieInternalWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Ib29kaWVJbnRlcm5hbFdyaXRlU3RhdHVzLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `43.37% <0.00%> (-0.15%)` | :arrow_down: | | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `62.36% <0.00%> (-2.39%)` | :arrow_down: | | [...n/java/org/apache/hudi/internal/DefaultSource.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmsyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2ludGVybmFsL0RlZmF1bHRTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/spark3/internal/DefaultSource.java](https://codecov.io/gh/apach
[GitHub] [hudi] hudi-bot edited a comment on pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 2cf38df103e1435c4239824f4908520727e38f69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=958) * 01fa47d89991e0de17d8cbe50e6a8aae04ea7690 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=984) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
codecov-commenter edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816086 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3247](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5b5c7c2) into [master](https://codecov.io/gh/apache/hudi/commit/50c2b76d725a71608a38217370b1ac45cedae405?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (50c2b76) will **decrease** coverage by `1.76%`. > The diff coverage is `62.06%`. [](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3247 +/- ## - Coverage 47.78% 46.01% -1.77% + Complexity 5557 4763 -794 Files 936 832 -104 Lines 4159638058-3538 Branches 4185 3809 -376 - Hits 1987717514-2363 + Misses1994918920-1029 + Partials 1770 1624 -146 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `23.00% <0.00%> (-11.53%)` | :arrow_down: | | hudicommon | `48.67% <26.66%> (-0.03%)` | :arrow_down: | | hudiflink | `59.36% <ø> (ø)` | | | hudihadoopmr | `52.02% <ø> (ø)` | | | hudisparkdatasource | `67.12% <73.52%> (-0.26%)` | :arrow_down: | | hudisync | `55.97% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/HoodieInternalWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Ib29kaWVJbnRlcm5hbFdyaXRlU3RhdHVzLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `43.37% <0.00%> (-0.15%)` | :arrow_down: | | [...pache/hudi/common/table/HoodieTableMetaClient.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL0hvb2RpZVRhYmxlTWV0YUNsaWVudC5qYXZh) | `62.36% <0.00%> (-2.39%)` | :arrow_down: | | [...n/java/org/apache/hudi/internal/DefaultSource.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmsyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2ludGVybmFsL0RlZmF1bHRTb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/spark3/internal/DefaultSource.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmszL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3N
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382467#comment-17382467 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 2cf38df103e1435c4239824f4908520727e38f69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=958) * 01fa47d89991e0de17d8cbe50e6a8aae04ea7690 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3282: [HUDI-2176, 2178, 2179] Virtual keys support to COW table
hudi-bot edited a comment on pull request #3282: URL: https://github.com/apache/hudi/pull/3282#issuecomment-880706639 ## CI report: * 2cf38df103e1435c4239824f4908520727e38f69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=958) * 01fa47d89991e0de17d8cbe50e6a8aae04ea7690 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382464#comment-17382464 ] ASF GitHub Bot commented on HUDI-2161: -- codecov-commenter edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816086 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
codecov-commenter edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816086 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382463#comment-17382463 ] ASF GitHub Bot commented on HUDI-2161: -- hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 5e7e02ec3da3137c31e2124c88cff815bc299875 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=980) * 5b5c7c2fcdff2a5f5e0737da8c9d03da83c4a65c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=983) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 5e7e02ec3da3137c31e2124c88cff815bc299875 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=980) * 5b5c7c2fcdff2a5f5e0737da8c9d03da83c4a65c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=983) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1884) MergeInto Support Partial Update For COW
[
https://issues.apache.org/jira/browse/HUDI-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382462#comment-17382462
]
ASF GitHub Bot commented on HUDI-1884:
--
pengzhiwei2018 merged pull request #3154:
URL: https://github.com/apache/hudi/pull/3154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MergeInto Support Partial Update For COW
> -
>
> Key: HUDI-1884
> URL: https://issues.apache.org/jira/browse/HUDI-1884
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
>
> Support partial update for merge-into statement, just like this:
> {code:java}
> merge into h0
> using s0
> on s0.id = h0.id
> when matched then update set price = s0.price + 10
> when not matched then insert *{code}
> Allow to update partial fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[hudi] branch master updated (af837d2 -> 572a214)
This is an automated email from the ASF dual-hosted git repository. zhiwei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from af837d2 [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp (#2438) add 572a214 [HUDI-1884] MergeInto Support Partial Update For COW (#3154) No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/hudi/HoodieSqlUtils.scala | 14 ++- .../spark/sql/hudi/analysis/HoodieAnalysis.scala | 49 +++- .../hudi/command/MergeIntoHoodieTableCommand.scala | 12 +- .../sql/hudi/TestPartialUpdateForMergeInto.scala | 137 + .../apache/spark/sql/adapter/Spark2Adapter.scala | 2 +- .../apache/spark/sql/adapter/Spark3Adapter.scala | 2 +- 6 files changed, 200 insertions(+), 16 deletions(-) create mode 100644 hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestPartialUpdateForMergeInto.scala
[GitHub] [hudi] pengzhiwei2018 merged pull request #3154: [HUDI-1884] MergeInto Support Partial Update For COW
pengzhiwei2018 merged pull request #3154: URL: https://github.com/apache/hudi/pull/3154 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1160) Support update partial fields for CoW table
[ https://issues.apache.org/jira/browse/HUDI-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382457#comment-17382457 ] ASF GitHub Bot commented on HUDI-1160: -- hudi-bot edited a comment on pull request #2666: URL: https://github.com/apache/hudi/pull/2666#issuecomment-881826594 ## CI report: * 7eeba905b81188f272e58401932887dbf93dc4fe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=981) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support update partial fields for CoW table > --- > > Key: HUDI-1160 > URL: https://issues.apache.org/jira/browse/HUDI-1160 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Reporter: leesf >Assignee: leesf >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #2666: [HUDI-1160] Support update partial fields for CoW table
hudi-bot edited a comment on pull request #2666: URL: https://github.com/apache/hudi/pull/2666#issuecomment-881826594 ## CI report: * 7eeba905b81188f272e58401932887dbf93dc4fe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=981) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1160) Support update partial fields for CoW table
[ https://issues.apache.org/jira/browse/HUDI-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382456#comment-17382456 ] ASF GitHub Bot commented on HUDI-1160: -- hudi-bot commented on pull request #2666: URL: https://github.com/apache/hudi/pull/2666#issuecomment-881826594 ## CI report: * 7eeba905b81188f272e58401932887dbf93dc4fe UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support update partial fields for CoW table > --- > > Key: HUDI-1160 > URL: https://issues.apache.org/jira/browse/HUDI-1160 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Reporter: leesf >Assignee: leesf >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382455#comment-17382455 ] ASF GitHub Bot commented on HUDI-2161: -- codecov-commenter edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816086 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3247](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e7e02e) into [master](https://codecov.io/gh/apache/hudi/commit/50c2b76d725a71608a38217370b1ac45cedae405?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (50c2b76) will **decrease** coverage by `32.03%`. > The diff coverage is `0.00%`. > :exclamation: Current head 5e7e02e differs from pull request most recent head 5b5c7c2. Consider uploading reports for the commit 5b5c7c2 to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3247 +/- ## = - Coverage 47.78% 15.75% -32.04% + Complexity 5557 493 -5064 = Files 936 284 -652 Lines 4159611832-29764 Branches 4185 981 -3204 = - Hits 19877 1864-18013 + Misses19949 9805-10144 + Partials 1770 163 -1607 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/HoodieInternalWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Ib29kaWVJbnRlcm5hbFdyaXRlU3RhdHVzLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.52%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` |
[GitHub] [hudi] hudi-bot commented on pull request #2666: [HUDI-1160] Support update partial fields for CoW table
hudi-bot commented on pull request #2666: URL: https://github.com/apache/hudi/pull/2666#issuecomment-881826594 ## CI report: * 7eeba905b81188f272e58401932887dbf93dc4fe UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
codecov-commenter edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816086 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3247](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e7e02e) into [master](https://codecov.io/gh/apache/hudi/commit/50c2b76d725a71608a38217370b1ac45cedae405?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (50c2b76) will **decrease** coverage by `32.03%`. > The diff coverage is `0.00%`. > :exclamation: Current head 5e7e02e differs from pull request most recent head 5b5c7c2. Consider uploading reports for the commit 5b5c7c2 to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3247 +/- ## = - Coverage 47.78% 15.75% -32.04% + Complexity 5557 493 -5064 = Files 936 284 -652 Lines 4159611832-29764 Branches 4185 981 -3204 = - Hits 19877 1864-18013 + Misses19949 9805-10144 + Partials 1770 163 -1607 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.26% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/HoodieInternalWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Ib29kaWVJbnRlcm5hbFdyaXRlU3RhdHVzLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.52%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software
[jira] [Commented] (HUDI-1160) Support update partial fields for CoW table
[ https://issues.apache.org/jira/browse/HUDI-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382454#comment-17382454 ] ASF GitHub Bot commented on HUDI-1160: -- liujinhui1994 commented on pull request #2666: URL: https://github.com/apache/hudi/pull/2666#issuecomment-881826299 @nsivabalan hello, I see that some partners have already suggested this idea. [HUDI-1884] MergeInto Support Partial Update For COW Are we still necessary to carry out this PR, if so, I will continue to deal with it and do a good job to improve -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support update partial fields for CoW table > --- > > Key: HUDI-1160 > URL: https://issues.apache.org/jira/browse/HUDI-1160 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core >Reporter: leesf >Assignee: leesf >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] liujinhui1994 commented on pull request #2666: [HUDI-1160] Support update partial fields for CoW table
liujinhui1994 commented on pull request #2666: URL: https://github.com/apache/hudi/pull/2666#issuecomment-881826299 @nsivabalan hello, I see that some partners have already suggested this idea. [HUDI-1884] MergeInto Support Partial Update For COW Are we still necessary to carry out this PR, if so, I will continue to deal with it and do a good job to improve -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382452#comment-17382452
]
ASF GitHub Bot commented on HUDI-1860:
--
nsivabalan commented on a change in pull request #3184:
URL: https://github.com/apache/hudi/pull/3184#discussion_r671607820
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##
@@ -475,8 +485,8 @@ public void refreshTimeline() throws IOException {
LOG.warn("Some records failed to be merged but forcing commit since
commitOnErrors set. Errors/Total="
+ totalErrorRecords + "/" + totalRecords);
}
-
- boolean success = writeClient.commit(instantTime, writeStatusRDD,
Option.of(checkpointCommitMetadata));
+ String commitActionType = CommitUtils.getCommitActionType(cfg.operation,
HoodieTableType.valueOf(cfg.tableType));
Review comment:
I meant to fix it in this PR itself. just one line java docs to each
method. @codope : Can you coordinate w/ samrat and fix the docs.
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -1695,6 +1695,54 @@ public void
testJdbcSourceIncrementalFetchInContinuousMode() {
}
}
+ @Test
+ public void testInsertOverwrite() throws Exception {
+String tableBasePath = dfsBasePath + "/insert_overwrite";
+// Initial insert
+HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath,
WriteOperationType.INSERT);
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// No new data => no commits.
+cfg.sourceLimit = 0;
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// insert overwrite
+cfg.sourceLimit = 1000;
+cfg.operation = WriteOperationType.INSERT_OVERWRITE;
Review comment:
:) I get it. Guess that test also needs some fixing.
Basically we wanna verify that insert_overwrite does not overwrite
mismatched partitions. So, better to cover it as part of tests. If you feel, it
might take lot of time to get this in, its ok. Do file a ticket. One of us from
the community will follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add INSERT_OVERWRITE support to DeltaStreamer
> -
>
> Key: HUDI-1860
> URL: https://issues.apache.org/jira/browse/HUDI-1860
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: Sagar Sumit
>Assignee: Samrat Deb
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> As discussed in [this
> RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller],
> having full fetch mode use the inser_overwrite to write to sync would be
> better as it can handle schema changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
nsivabalan commented on a change in pull request #3184:
URL: https://github.com/apache/hudi/pull/3184#discussion_r671607820
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##
@@ -475,8 +485,8 @@ public void refreshTimeline() throws IOException {
LOG.warn("Some records failed to be merged but forcing commit since
commitOnErrors set. Errors/Total="
+ totalErrorRecords + "/" + totalRecords);
}
-
- boolean success = writeClient.commit(instantTime, writeStatusRDD,
Option.of(checkpointCommitMetadata));
+ String commitActionType = CommitUtils.getCommitActionType(cfg.operation,
HoodieTableType.valueOf(cfg.tableType));
Review comment:
I meant to fix it in this PR itself. just one line java docs to each
method. @codope : Can you coordinate w/ samrat and fix the docs.
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -1695,6 +1695,54 @@ public void
testJdbcSourceIncrementalFetchInContinuousMode() {
}
}
+ @Test
+ public void testInsertOverwrite() throws Exception {
+String tableBasePath = dfsBasePath + "/insert_overwrite";
+// Initial insert
+HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath,
WriteOperationType.INSERT);
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// No new data => no commits.
+cfg.sourceLimit = 0;
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// insert overwrite
+cfg.sourceLimit = 1000;
+cfg.operation = WriteOperationType.INSERT_OVERWRITE;
Review comment:
:) I get it. Guess that test also needs some fixing.
Basically we wanna verify that insert_overwrite does not overwrite
mismatched partitions. So, better to cover it as part of tests. If you feel, it
might take lot of time to get this in, its ok. Do file a ticket. One of us from
the community will follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382451#comment-17382451 ] ASF GitHub Bot commented on HUDI-2137: -- hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * b2ea6fd20027717598af5310c1d626100e212e8a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=979) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make hudi examples runnable locally > --- > > Key: HUDI-2137 > URL: https://issues.apache.org/jira/browse/HUDI-2137 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * b2ea6fd20027717598af5310c1d626100e212e8a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=979) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382450#comment-17382450 ] ASF GitHub Bot commented on HUDI-1447: -- liujinhui1994 commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-881820983 @nsivabalan Thank you for your concern and patience to help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DeltaStreamer kafka source supports consuming from specified timestamp > -- > > Key: HUDI-1447 > URL: https://issues.apache.org/jira/browse/HUDI-1447 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer >Reporter: wangxianghu#1 >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:high, user-support-issues > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] liujinhui1994 commented on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
liujinhui1994 commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-881820983 @nsivabalan Thank you for your concern and patience to help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382449#comment-17382449 ] ASF GitHub Bot commented on HUDI-1447: -- nsivabalan merged pull request #2438: URL: https://github.com/apache/hudi/pull/2438 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DeltaStreamer kafka source supports consuming from specified timestamp > -- > > Key: HUDI-1447 > URL: https://issues.apache.org/jira/browse/HUDI-1447 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer >Reporter: wangxianghu#1 >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:high, user-support-issues > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[hudi] branch master updated: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp (#2438)
This is an automated email from the ASF dual-hosted git repository.
sivabalan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new af837d2 [HUDI-1447] DeltaStreamer kafka source supports consuming
from specified timestamp (#2438)
af837d2 is described below
commit af837d2f1825d14ae8403b2290cf5eab39780343
Author: liujinhui <965147...@qq.com>
AuthorDate: Sat Jul 17 12:31:06 2021 +0800
[HUDI-1447] DeltaStreamer kafka source supports consuming from specified
timestamp (#2438)
---
.../hudi/utilities/deltastreamer/DeltaSync.java| 15 ++-
.../hudi/utilities/sources/AvroKafkaSource.java| 5 +-
.../hudi/utilities/sources/JsonKafkaSource.java| 5 +-
.../utilities/sources/helpers/KafkaOffsetGen.java | 103 ++---
.../functional/TestHoodieDeltaStreamer.java| 56 ---
.../hudi/utilities/sources/TestKafkaSource.java| 15 +--
.../sources/helpers/TestKafkaOffsetGen.java| 35 +--
7 files changed, 172 insertions(+), 62 deletions(-)
diff --git
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
index 7742e8e..9d445dc 100644
---
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
+++
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
@@ -38,6 +38,7 @@ import org.apache.hudi.common.table.timeline.HoodieInstant;
import org.apache.hudi.common.table.timeline.HoodieTimeline;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.StringUtils;
import org.apache.hudi.common.util.ValidationUtils;
import org.apache.hudi.common.util.collection.Pair;
import org.apache.hudi.config.HoodieClusteringConfig;
@@ -59,6 +60,7 @@ import
org.apache.hudi.utilities.schema.DelegatingSchemaProvider;
import org.apache.hudi.utilities.schema.SchemaProvider;
import org.apache.hudi.utilities.schema.SchemaSet;
import org.apache.hudi.utilities.sources.InputBatch;
+import org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen;
import org.apache.hudi.utilities.transform.Transformer;
import com.codahale.metrics.Timer;
@@ -318,13 +320,12 @@ public class DeltaSync implements Serializable {
if (lastCommit.isPresent()) {
HoodieCommitMetadata commitMetadata = HoodieCommitMetadata
.fromBytes(commitTimelineOpt.get().getInstantDetails(lastCommit.get()).get(),
HoodieCommitMetadata.class);
-if (cfg.checkpoint != null &&
!cfg.checkpoint.equals(commitMetadata.getMetadata(CHECKPOINT_RESET_KEY))) {
+if (cfg.checkpoint != null &&
(StringUtils.isNullOrEmpty(commitMetadata.getMetadata(CHECKPOINT_RESET_KEY))
+||
!cfg.checkpoint.equals(commitMetadata.getMetadata(CHECKPOINT_RESET_KEY {
resumeCheckpointStr = Option.of(cfg.checkpoint);
-} else if (commitMetadata.getMetadata(CHECKPOINT_KEY) != null) {
+} else if
(!StringUtils.isNullOrEmpty(commitMetadata.getMetadata(CHECKPOINT_KEY))) {
//if previous checkpoint is an empty string, skip resume use
Option.empty()
- if (!commitMetadata.getMetadata(CHECKPOINT_KEY).isEmpty()) {
-resumeCheckpointStr =
Option.of(commitMetadata.getMetadata(CHECKPOINT_KEY));
- }
+ resumeCheckpointStr =
Option.of(commitMetadata.getMetadata(CHECKPOINT_KEY));
} else if (commitMetadata.getOperationType() ==
WriteOperationType.CLUSTER) {
// incase of CLUSTER commit, no checkpoint will be available in
metadata.
resumeCheckpointStr = Option.empty();
@@ -336,6 +337,10 @@ public class DeltaSync implements Serializable {
+
commitTimelineOpt.get().getInstants().collect(Collectors.toList()) + ",
CommitMetadata="
+ commitMetadata.toJsonString());
}
+// KAFKA_CHECKPOINT_TYPE will be honored only for first batch.
+if
(!StringUtils.isNullOrEmpty(commitMetadata.getMetadata(CHECKPOINT_RESET_KEY))) {
+ props.remove(KafkaOffsetGen.Config.KAFKA_CHECKPOINT_TYPE.key());
+}
}
} else {
String partitionColumns =
HoodieWriterUtils.getPartitionColumns(keyGenerator);
diff --git
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java
index 4cea13d..500c412 100644
---
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java
+++
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java
@@ -40,9 +40,6 @@ import org.apache.spark.streaming.kafka010.KafkaUtils;
import org.apache.spark.streaming.kafka010.LocationStrategies;
import org.apache.spark.streaming.kafka0
[GitHub] [hudi] nsivabalan merged pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
nsivabalan merged pull request #2438: URL: https://github.com/apache/hudi/pull/2438 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382447#comment-17382447 ] ASF GitHub Bot commented on HUDI-2161: -- codecov-commenter edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816086 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3247](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5b5c7c2) into [master](https://codecov.io/gh/apache/hudi/commit/50c2b76d725a71608a38217370b1ac45cedae405?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (50c2b76) will **decrease** coverage by `44.95%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3247 +/- ## - Coverage 47.78% 2.83% -44.96% + Complexity 5557 85 -5472 Files 936 284 -652 Lines 41596 11832-29764 Branches 4185 981 -3204 - Hits 19877 335-19542 + Misses19949 11471 -8478 + Partials 1770 26 -1744 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-50.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/HoodieInternalWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Ib29kaWVJbnRlcm5hbFdyaXRlU3RhdHVzLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.52%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comme
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[ https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382448#comment-17382448 ] ASF GitHub Bot commented on HUDI-1447: -- nsivabalan commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-881817246 Appreciate your perseverance in addressing all the feedback. You are the best! :) Thanks for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DeltaStreamer kafka source supports consuming from specified timestamp > -- > > Key: HUDI-1447 > URL: https://issues.apache.org/jira/browse/HUDI-1447 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer >Reporter: wangxianghu#1 >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:high, user-support-issues > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
codecov-commenter edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816086 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3247](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5b5c7c2) into [master](https://codecov.io/gh/apache/hudi/commit/50c2b76d725a71608a38217370b1ac45cedae405?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (50c2b76) will **decrease** coverage by `44.95%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3247 +/- ## - Coverage 47.78% 2.83% -44.96% + Complexity 5557 85 -5472 Files 936 284 -652 Lines 41596 11832-29764 Branches 4185 981 -3204 - Hits 19877 335-19542 + Misses19949 11471 -8478 + Partials 1770 26 -1744 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-50.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/HoodieInternalWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Ib29kaWVJbnRlcm5hbFdyaXRlU3RhdHVzLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.52%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi
[GitHub] [hudi] nsivabalan commented on pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
nsivabalan commented on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-881817246 Appreciate your perseverance in addressing all the feedback. You are the best! :) Thanks for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1447) DeltaStreamer kafka source supports consuming from specified timestamp
[
https://issues.apache.org/jira/browse/HUDI-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382446#comment-17382446
]
ASF GitHub Bot commented on HUDI-1447:
--
nsivabalan commented on a change in pull request #2438:
URL: https://github.com/apache/hudi/pull/2438#discussion_r671607330
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##
@@ -212,6 +234,9 @@ public KafkaOffsetGen(TypedProperties props) {
Set topicPartitions = partitionInfoList.stream()
.map(x -> new TopicPartition(x.topic(),
x.partition())).collect(Collectors.toSet());
+ if (Config.KAFKA_CHECKPOINT_TYPE_TIMESTAMP.equals(kafkaCheckpointType)
&& isValidCheckpointType(lastCheckpointStr)) {
+lastCheckpointStr = getOffsetsByTimestamp(consumer, partitionInfoList,
topicPartitions, topicName, Long.parseLong(lastCheckpointStr.get()));
+ }
Review comment:
ok, I get it now. makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DeltaStreamer kafka source supports consuming from specified timestamp
> --
>
> Key: HUDI-1447
> URL: https://issues.apache.org/jira/browse/HUDI-1447
> Project: Apache Hudi
> Issue Type: New Feature
> Components: DeltaStreamer
>Reporter: wangxianghu#1
>Assignee: liujinhui
>Priority: Major
> Labels: pull-request-available, sev:high, user-support-issues
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #2438: [HUDI-1447] DeltaStreamer kafka source supports consuming from specified timestamp
nsivabalan commented on a change in pull request #2438:
URL: https://github.com/apache/hudi/pull/2438#discussion_r671607330
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##
@@ -212,6 +234,9 @@ public KafkaOffsetGen(TypedProperties props) {
Set topicPartitions = partitionInfoList.stream()
.map(x -> new TopicPartition(x.topic(),
x.partition())).collect(Collectors.toSet());
+ if (Config.KAFKA_CHECKPOINT_TYPE_TIMESTAMP.equals(kafkaCheckpointType)
&& isValidCheckpointType(lastCheckpointStr)) {
+lastCheckpointStr = getOffsetsByTimestamp(consumer, partitionInfoList,
topicPartitions, topicName, Long.parseLong(lastCheckpointStr.get()));
+ }
Review comment:
ok, I get it now. makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382445#comment-17382445 ] ASF GitHub Bot commented on HUDI-2161: -- hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 5e7e02ec3da3137c31e2124c88cff815bc299875 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=980) * 5b5c7c2fcdff2a5f5e0737da8c9d03da83c4a65c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 5e7e02ec3da3137c31e2124c88cff815bc299875 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=980) * 5b5c7c2fcdff2a5f5e0737da8c9d03da83c4a65c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382444#comment-17382444 ] ASF GitHub Bot commented on HUDI-2161: -- nsivabalan commented on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816784 Sorry I had squashed all commits to one so that its easier for me to rebase. I had 10+ files in conflict when I rebased w/ master. Also, I did rename quite a few files and moved some of them(reuse spark serDe class), and so went ahead and squashed it. sorry if you were planning to review just the latest commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
nsivabalan commented on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816784 Sorry I had squashed all commits to one so that its easier for me to rebase. I had 10+ files in conflict when I rebased w/ master. Also, I did rename quite a few files and moved some of them(reuse spark serDe class), and so went ahead and squashed it. sorry if you were planning to review just the latest commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382443#comment-17382443 ] ASF GitHub Bot commented on HUDI-2161: -- hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=978) * 5e7e02ec3da3137c31e2124c88cff815bc299875 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=980) * 5b5c7c2fcdff2a5f5e0737da8c9d03da83c4a65c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=978) * 5e7e02ec3da3137c31e2124c88cff815bc299875 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=980) * 5b5c7c2fcdff2a5f5e0737da8c9d03da83c4a65c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[
https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382438#comment-17382438
]
ASF GitHub Bot commented on HUDI-2161:
--
nsivabalan commented on a change in pull request #3247:
URL: https://github.com/apache/hudi/pull/3247#discussion_r671606366
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##
@@ -128,14 +128,35 @@ object HoodieSparkSqlWriter {
.setPayloadClassName(hoodieConfig.getString(PAYLOAD_CLASS_OPT_KEY))
.setPreCombineField(hoodieConfig.getStringOrDefault(PRECOMBINE_FIELD_OPT_KEY,
null))
.setPartitionColumns(partitionColumns)
+
.setPopulateMetaColumns(parameters.getOrElse(HoodieTableConfig.HOODIE_POPULATE_META_COLUMNS.key(),
HoodieTableConfig.HOODIE_POPULATE_META_COLUMNS.defaultValue()).toBoolean)
.initTable(sparkContext.hadoopConfiguration, path.get)
tableConfig = tableMetaClient.getTableConfig
+ } else {
+// validate table properties
+val tableMetaClient =
HoodieTableMetaClient.builder().setBasePath(path.get).setConf(sparkContext.hadoopConfiguration).build()
Review comment:
Added a private method here in HoodiesparkSqlWriter wrt params and add a
method in HoodieTableMetaclient to validate table properties.
##
File path:
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala
##
@@ -21,21 +21,24 @@ package org.apache.hudi
import org.apache.avro.Schema
import org.apache.avro.generic.GenericRecord
import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hudi.client.utils.SparkRowSerDe
Review comment:
Moved HoodiesparkUtils, SparkAdaptorSupport and SparkAdaptor from
hudi-spark module to hudi-spark-client module since we wanted to access
SparkAdaptor from within BuiltInKeygen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add support to disable meta column to BulkInsert Row Writer path
>
>
> Key: HUDI-2161
> URL: https://issues.apache.org/jira/browse/HUDI-2161
> Project: Apache Hudi
> Issue Type: Improvement
>Reporter: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
>
> Objective here is to disable all meta columns so as to avoid storage cost.
> Also, some benefits could be seen in write latency with row writer path as no
> special handling is required at RowCreateHandle layer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382439#comment-17382439 ] ASF GitHub Bot commented on HUDI-2161: -- codecov-commenter commented on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816086 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3247](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e7e02e) into [master](https://codecov.io/gh/apache/hudi/commit/50c2b76d725a71608a38217370b1ac45cedae405?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (50c2b76) will **decrease** coverage by `44.95%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3247 +/- ## - Coverage 47.78% 2.83% -44.96% + Complexity 5557 85 -5472 Files 936 284 -652 Lines 41596 11832-29764 Branches 4185 981 -3204 - Hits 19877 335-19542 + Misses19949 11471 -8478 + Partials 1770 26 -1744 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-50.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/HoodieInternalWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Ib29kaWVJbnRlcm5hbFdyaXRlU3RhdHVzLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.52%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_
[GitHub] [hudi] codecov-commenter commented on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
codecov-commenter commented on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-881816086 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3247](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e7e02e) into [master](https://codecov.io/gh/apache/hudi/commit/50c2b76d725a71608a38217370b1ac45cedae405?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (50c2b76) will **decrease** coverage by `44.95%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3247 +/- ## - Coverage 47.78% 2.83% -44.96% + Complexity 5557 85 -5472 Files 936 284 -652 Lines 41596 11832-29764 Branches 4185 981 -3204 - Hits 19877 335-19542 + Misses19949 11471 -8478 + Partials 1770 26 -1744 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (-50.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/HoodieInternalWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9Ib29kaWVJbnRlcm5hbFdyaXRlU3RhdHVzLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.52%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilit
[GitHub] [hudi] nsivabalan commented on a change in pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
nsivabalan commented on a change in pull request #3247:
URL: https://github.com/apache/hudi/pull/3247#discussion_r671606366
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##
@@ -128,14 +128,35 @@ object HoodieSparkSqlWriter {
.setPayloadClassName(hoodieConfig.getString(PAYLOAD_CLASS_OPT_KEY))
.setPreCombineField(hoodieConfig.getStringOrDefault(PRECOMBINE_FIELD_OPT_KEY,
null))
.setPartitionColumns(partitionColumns)
+
.setPopulateMetaColumns(parameters.getOrElse(HoodieTableConfig.HOODIE_POPULATE_META_COLUMNS.key(),
HoodieTableConfig.HOODIE_POPULATE_META_COLUMNS.defaultValue()).toBoolean)
.initTable(sparkContext.hadoopConfiguration, path.get)
tableConfig = tableMetaClient.getTableConfig
+ } else {
+// validate table properties
+val tableMetaClient =
HoodieTableMetaClient.builder().setBasePath(path.get).setConf(sparkContext.hadoopConfiguration).build()
Review comment:
Added a private method here in HoodiesparkSqlWriter wrt params and add a
method in HoodieTableMetaclient to validate table properties.
##
File path:
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala
##
@@ -21,21 +21,24 @@ package org.apache.hudi
import org.apache.avro.Schema
import org.apache.avro.generic.GenericRecord
import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hudi.client.utils.SparkRowSerDe
Review comment:
Moved HoodiesparkUtils, SparkAdaptorSupport and SparkAdaptor from
hudi-spark module to hudi-spark-client module since we wanted to access
SparkAdaptor from within BuiltInKeygen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382436#comment-17382436 ] ASF GitHub Bot commented on HUDI-2161: -- hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=978) * 5e7e02ec3da3137c31e2124c88cff815bc299875 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=980) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=978) * 5e7e02ec3da3137c31e2124c88cff815bc299875 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=980) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382435#comment-17382435 ] ASF GitHub Bot commented on HUDI-2161: -- hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=978) * 5e7e02ec3da3137c31e2124c88cff815bc299875 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=978) * 5e7e02ec3da3137c31e2124c88cff815bc299875 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382433#comment-17382433 ] ASF GitHub Bot commented on HUDI-2161: -- hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=978) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=978) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382432#comment-17382432 ] ASF GitHub Bot commented on HUDI-2137: -- codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make hudi examples runnable locally > --- > > Key: HUDI-2137 > URL: https://issues.apache.org/jira/browse/HUDI-2137 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382431#comment-17382431 ] ASF GitHub Bot commented on HUDI-2137: -- codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3244](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b2ea6fd) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **decrease** coverage by `16.78%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3244 +/- ## = - Coverage 44.10% 27.32% -16.79% + Complexity 5157 1292 -3865 = Files 936 386 -550 Lines 4162915336-26293 Branches 4189 1338 -2851 = - Hits 18362 4190-14172 + Misses2163810842-10796 + Partials 1629 304 -1325 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `20.91% <ø> (-13.55%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-50.86%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.26% <ø> (+50.14%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_med
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3244](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b2ea6fd) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **decrease** coverage by `16.78%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3244 +/- ## = - Coverage 44.10% 27.32% -16.79% + Complexity 5157 1292 -3865 = Files 936 386 -550 Lines 4162915336-26293 Branches 4189 1338 -2851 = - Hits 18362 4190-14172 + Misses2163810842-10796 + Partials 1629 304 -1325 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `20.91% <ø> (-13.55%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-50.86%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.26% <ø> (+50.14%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh)
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382430#comment-17382430 ] ASF GitHub Bot commented on HUDI-2137: -- hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * ad8dc0793252287cd160e9bca9cc678f172afd2d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=977) * b2ea6fd20027717598af5310c1d626100e212e8a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=979) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make hudi examples runnable locally > --- > > Key: HUDI-2137 > URL: https://issues.apache.org/jira/browse/HUDI-2137 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * ad8dc0793252287cd160e9bca9cc678f172afd2d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=977) * b2ea6fd20027717598af5310c1d626100e212e8a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=979) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382429#comment-17382429 ] ASF GitHub Bot commented on HUDI-2137: -- codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make hudi examples runnable locally > --- > > Key: HUDI-2137 > URL: https://issues.apache.org/jira/browse/HUDI-2137 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382428#comment-17382428 ] ASF GitHub Bot commented on HUDI-2137: -- codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3244](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b2ea6fd) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **decrease** coverage by `28.34%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3244 +/- ## = - Coverage 44.10% 15.75% -28.35% + Complexity 5157 493 -4664 = Files 936 284 -652 Lines 4162911828-29801 Branches 4189 981 -3208 = - Hits 18362 1864-16498 + Misses21638 9801-11837 + Partials 1629 163 -1466 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.47%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-50.86%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.26% <ø> (+50.14%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medi
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3244](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b2ea6fd) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **decrease** coverage by `28.34%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3244 +/- ## = - Coverage 44.10% 15.75% -28.35% + Complexity 5157 493 -4664 = Files 936 284 -652 Lines 4162911828-29801 Branches 4189 981 -3208 = - Hits 18362 1864-16498 + Misses21638 9801-11837 + Partials 1629 163 -1466 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.47%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-50.86%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.26% <ø> (+50.14%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) |
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382427#comment-17382427 ] ASF GitHub Bot commented on HUDI-2137: -- codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3244](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ad8dc07) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **increase** coverage by `15.14%`. > The diff coverage is `n/a`. > :exclamation: Current head ad8dc07 differs from pull request most recent head b2ea6fd. Consider uploading reports for the commit b2ea6fd to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3244 +/- ## = + Coverage 44.10% 59.24% +15.14% + Complexity 5157 1216 -3941 = Files 936 169 -767 Lines 41629 6557-35072 Branches 4189 687 -3502 = - Hits 18362 3885-14477 + Misses21638 2391-19247 + Partials 1629 281 -1348 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `66.30% <ø> (+31.83%)` | :arrow_up: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `51.13% <ø> (+42.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-78.45%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...org/apache/hudi/utilities/HDFSParquetImporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hERlNQYXJxdWV0SW1wb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-71.82%)` | :arrow_down: |
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[
https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382426#comment-17382426
]
ASF GitHub Bot commented on HUDI-2161:
--
nsivabalan commented on a change in pull request #3247:
URL: https://github.com/apache/hudi/pull/3247#discussion_r671600987
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##
@@ -128,14 +128,35 @@ object HoodieSparkSqlWriter {
.setPayloadClassName(hoodieConfig.getString(PAYLOAD_CLASS_OPT_KEY))
.setPreCombineField(hoodieConfig.getStringOrDefault(PRECOMBINE_FIELD_OPT_KEY,
null))
.setPartitionColumns(partitionColumns)
+
.setPopulateMetaColumns(parameters.getOrElse(HoodieTableConfig.HOODIE_POPULATE_META_COLUMNS.key(),
HoodieTableConfig.HOODIE_POPULATE_META_COLUMNS.defaultValue()).toBoolean)
.initTable(sparkContext.hadoopConfiguration, path.get)
tableConfig = tableMetaClient.getTableConfig
+ } else {
+// validate table properties
+val tableMetaClient =
HoodieTableMetaClient.builder().setBasePath(path.get).setConf(sparkContext.hadoopConfiguration).build()
Review comment:
I am thinking to add the validation within HoodieTableMetaClient.
Because, a) We don't instantiate WriteClient at all in row writer path as of
now. b) table properties are available when tableMetaclient is instantiated.
Even if we were to place it within WriteClient, we have to instantiate
Metaclient to read the table props. So, we might as well place the validation
method in HoodieTableMetaClient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add support to disable meta column to BulkInsert Row Writer path
>
>
> Key: HUDI-2161
> URL: https://issues.apache.org/jira/browse/HUDI-2161
> Project: Apache Hudi
> Issue Type: Improvement
>Reporter: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
>
> Objective here is to disable all meta columns so as to avoid storage cost.
> Also, some benefits could be seen in write latency with row writer path as no
> special handling is required at RowCreateHandle layer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3244](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ad8dc07) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **increase** coverage by `15.14%`. > The diff coverage is `n/a`. > :exclamation: Current head ad8dc07 differs from pull request most recent head b2ea6fd. Consider uploading reports for the commit b2ea6fd to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3244 +/- ## = + Coverage 44.10% 59.24% +15.14% + Complexity 5157 1216 -3941 = Files 936 169 -767 Lines 41629 6557-35072 Branches 4189 687 -3502 = - Hits 18362 3885-14477 + Misses21638 2391-19247 + Partials 1629 281 -1348 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `66.30% <ø> (+31.83%)` | :arrow_up: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `51.13% <ø> (+42.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-78.45%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...org/apache/hudi/utilities/HDFSParquetImporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hERlNQYXJxdWV0SW1wb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-71.82%)` | :arrow_down: | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#dif
[GitHub] [hudi] nsivabalan commented on a change in pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
nsivabalan commented on a change in pull request #3247:
URL: https://github.com/apache/hudi/pull/3247#discussion_r671600987
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##
@@ -128,14 +128,35 @@ object HoodieSparkSqlWriter {
.setPayloadClassName(hoodieConfig.getString(PAYLOAD_CLASS_OPT_KEY))
.setPreCombineField(hoodieConfig.getStringOrDefault(PRECOMBINE_FIELD_OPT_KEY,
null))
.setPartitionColumns(partitionColumns)
+
.setPopulateMetaColumns(parameters.getOrElse(HoodieTableConfig.HOODIE_POPULATE_META_COLUMNS.key(),
HoodieTableConfig.HOODIE_POPULATE_META_COLUMNS.defaultValue()).toBoolean)
.initTable(sparkContext.hadoopConfiguration, path.get)
tableConfig = tableMetaClient.getTableConfig
+ } else {
+// validate table properties
+val tableMetaClient =
HoodieTableMetaClient.builder().setBasePath(path.get).setConf(sparkContext.hadoopConfiguration).build()
Review comment:
I am thinking to add the validation within HoodieTableMetaClient.
Because, a) We don't instantiate WriteClient at all in row writer path as of
now. b) table properties are available when tableMetaclient is instantiated.
Even if we were to place it within WriteClient, we have to instantiate
Metaclient to read the table props. So, we might as well place the validation
method in HoodieTableMetaClient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382423#comment-17382423 ] ASF GitHub Bot commented on HUDI-2137: -- codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3244](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ad8dc07) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **increase** coverage by `7.02%`. > The diff coverage is `n/a`. > :exclamation: Current head ad8dc07 differs from pull request most recent head b2ea6fd. Consider uploading reports for the commit b2ea6fd to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3244 +/- ## + Coverage 44.10% 51.13% +7.02% + Complexity 5157 417-4740 Files 936 67 -869 Lines 41629 3049 -38580 Branches 4189 330-3859 - Hits 18362 1559 -16803 + Misses21638 1350 -20288 + Partials 1629 140-1489 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `?` | | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `51.13% <ø> (+42.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-78.45%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...org/apache/hudi/utilities/HDFSParquetImporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hERlNQYXJxdWV0SW1wb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-71.82%)` | :arrow_down: | | [...he/hudi/utilities/transform/AWSDms
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3244](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ad8dc07) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **increase** coverage by `7.02%`. > The diff coverage is `n/a`. > :exclamation: Current head ad8dc07 differs from pull request most recent head b2ea6fd. Consider uploading reports for the commit b2ea6fd to get more accurate results [](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3244 +/- ## + Coverage 44.10% 51.13% +7.02% + Complexity 5157 417-4740 Files 936 67 -869 Lines 41629 3049 -38580 Branches 4189 330-3859 - Hits 18362 1559 -16803 + Misses21638 1350 -20288 + Partials 1629 140-1489 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `?` | | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `?` | | | huditimelineservice | `?` | | | hudiutilities | `51.13% <ø> (+42.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-78.45%)` | :arrow_down: | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | :arrow_down: | | [...org/apache/hudi/utilities/HDFSParquetImporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hERlNQYXJxdWV0SW1wb3J0ZXIuamF2YQ==) | `0.00% <0.00%> (-71.82%)` | :arrow_down: | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcm
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382421#comment-17382421
]
ASF GitHub Bot commented on HUDI-1860:
--
Samrat002 commented on a change in pull request #3184:
URL: https://github.com/apache/hudi/pull/3184#discussion_r671598590
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -1695,6 +1695,54 @@ public void
testJdbcSourceIncrementalFetchInContinuousMode() {
}
}
+ @Test
+ public void testInsertOverwrite() throws Exception {
+String tableBasePath = dfsBasePath + "/insert_overwrite";
+// Initial insert
+HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath,
WriteOperationType.INSERT);
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// No new data => no commits.
+cfg.sourceLimit = 0;
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// insert overwrite
+cfg.sourceLimit = 1000;
+cfg.operation = WriteOperationType.INSERT_OVERWRITE;
Review comment:
hi @nsivabalan ,
for the 2nd point
do we need to cover the test senerio here, because I see test
(https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientOnCopyOnWriteStorage.java#L1227)
This test case is to check Deltastreamer when `op` is `INSERT_OVERWRITE`.
Moreover, it is using the same `SparkRDDWriteClient`.
please suggest if m looking at the correct test!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add INSERT_OVERWRITE support to DeltaStreamer
> -
>
> Key: HUDI-1860
> URL: https://issues.apache.org/jira/browse/HUDI-1860
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: Sagar Sumit
>Assignee: Samrat Deb
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> As discussed in [this
> RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller],
> having full fetch mode use the inser_overwrite to write to sync would be
> better as it can handle schema changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382422#comment-17382422 ] ASF GitHub Bot commented on HUDI-2137: -- codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3244](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b2ea6fd) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **decrease** coverage by `41.27%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3244 +/- ## - Coverage 44.10% 2.83% -41.28% + Complexity 5157 85 -5072 Files 936 284 -652 Lines 41629 11828-29801 Branches 4189 981 -3208 - Hits 18362 335-18027 + Misses21638 11467-10171 + Partials 1629 26 -1603 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.47%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-50.86%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&u
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
codecov-commenter edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876514322 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3244](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b2ea6fd) into [master](https://codecov.io/gh/apache/hudi/commit/d024439764ceeca6366cb33689b729a1c69a6272?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d024439) will **decrease** coverage by `41.27%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3244 +/- ## - Coverage 44.10% 2.83% -41.28% + Complexity 5157 85 -5072 Files 936 284 -652 Lines 41629 11828-29801 Branches 4189 981 -3208 - Hits 18362 335-18027 + Misses21638 11467-10171 + Partials 1629 26 -1603 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <ø> (-34.47%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-50.86%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `9.11% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3244?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3244/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :
[GitHub] [hudi] Samrat002 commented on a change in pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
Samrat002 commented on a change in pull request #3184:
URL: https://github.com/apache/hudi/pull/3184#discussion_r671598590
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -1695,6 +1695,54 @@ public void
testJdbcSourceIncrementalFetchInContinuousMode() {
}
}
+ @Test
+ public void testInsertOverwrite() throws Exception {
+String tableBasePath = dfsBasePath + "/insert_overwrite";
+// Initial insert
+HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath,
WriteOperationType.INSERT);
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// No new data => no commits.
+cfg.sourceLimit = 0;
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// insert overwrite
+cfg.sourceLimit = 1000;
+cfg.operation = WriteOperationType.INSERT_OVERWRITE;
Review comment:
hi @nsivabalan ,
for the 2nd point
do we need to cover the test senerio here, because I see test
(https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientOnCopyOnWriteStorage.java#L1227)
This test case is to check Deltastreamer when `op` is `INSERT_OVERWRITE`.
Moreover, it is using the same `SparkRDDWriteClient`.
please suggest if m looking at the correct test!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382418#comment-17382418 ] ASF GitHub Bot commented on HUDI-2137: -- hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * ad8dc0793252287cd160e9bca9cc678f172afd2d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=977) * b2ea6fd20027717598af5310c1d626100e212e8a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make hudi examples runnable locally > --- > > Key: HUDI-2137 > URL: https://issues.apache.org/jira/browse/HUDI-2137 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * ad8dc0793252287cd160e9bca9cc678f172afd2d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=977) * b2ea6fd20027717598af5310c1d626100e212e8a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382416#comment-17382416 ] ASF GitHub Bot commented on HUDI-2137: -- hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * 3822f58217e171cbf3ad513b4700415496520dff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=924) * ad8dc0793252287cd160e9bca9cc678f172afd2d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=977) * b2ea6fd20027717598af5310c1d626100e212e8a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make hudi examples runnable locally > --- > > Key: HUDI-2137 > URL: https://issues.apache.org/jira/browse/HUDI-2137 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382417#comment-17382417 ] ASF GitHub Bot commented on HUDI-2161: -- hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * e32037e79596e3bf19415d1af107850643ee9ee5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=976) * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=978) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3247: [HUDI-2161] Adding support to disable meta columns with bulk insert operation
hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * e32037e79596e3bf19415d1af107850643ee9ee5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=976) * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=978) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * 3822f58217e171cbf3ad513b4700415496520dff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=924) * ad8dc0793252287cd160e9bca9cc678f172afd2d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=977) * b2ea6fd20027717598af5310c1d626100e212e8a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382415#comment-17382415
]
ASF GitHub Bot commented on HUDI-1860:
--
Samrat002 commented on a change in pull request #3184:
URL: https://github.com/apache/hudi/pull/3184#discussion_r671598590
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -1695,6 +1695,54 @@ public void
testJdbcSourceIncrementalFetchInContinuousMode() {
}
}
+ @Test
+ public void testInsertOverwrite() throws Exception {
+String tableBasePath = dfsBasePath + "/insert_overwrite";
+// Initial insert
+HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath,
WriteOperationType.INSERT);
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// No new data => no commits.
+cfg.sourceLimit = 0;
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// insert overwrite
+cfg.sourceLimit = 1000;
+cfg.operation = WriteOperationType.INSERT_OVERWRITE;
Review comment:
hi @nsivabalan ,
for the 2nd point
do we need to cover the test case here, because I see test
(https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientOnCopyOnWriteStorage.java#L1227)
This test case is to check Deltastreamer when `op` is `INSERT_OVERWRITE`.
Moreover, it is using the same `SparkRDDWriteClient`. I feel it would be
redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add INSERT_OVERWRITE support to DeltaStreamer
> -
>
> Key: HUDI-1860
> URL: https://issues.apache.org/jira/browse/HUDI-1860
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: Sagar Sumit
>Assignee: Samrat Deb
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> As discussed in [this
> RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller],
> having full fetch mode use the inser_overwrite to write to sync would be
> better as it can handle schema changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] Samrat002 commented on a change in pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
Samrat002 commented on a change in pull request #3184:
URL: https://github.com/apache/hudi/pull/3184#discussion_r671598590
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -1695,6 +1695,54 @@ public void
testJdbcSourceIncrementalFetchInContinuousMode() {
}
}
+ @Test
+ public void testInsertOverwrite() throws Exception {
+String tableBasePath = dfsBasePath + "/insert_overwrite";
+// Initial insert
+HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath,
WriteOperationType.INSERT);
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// No new data => no commits.
+cfg.sourceLimit = 0;
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// insert overwrite
+cfg.sourceLimit = 1000;
+cfg.operation = WriteOperationType.INSERT_OVERWRITE;
Review comment:
hi @nsivabalan ,
for the 2nd point
do we need to cover the test case here, because I see test
(https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientOnCopyOnWriteStorage.java#L1227)
This test case is to check Deltastreamer when `op` is `INSERT_OVERWRITE`.
Moreover, it is using the same `SparkRDDWriteClient`. I feel it would be
redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382414#comment-17382414
]
ASF GitHub Bot commented on HUDI-1860:
--
Samrat002 commented on a change in pull request #3184:
URL: https://github.com/apache/hudi/pull/3184#discussion_r671598590
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -1695,6 +1695,54 @@ public void
testJdbcSourceIncrementalFetchInContinuousMode() {
}
}
+ @Test
+ public void testInsertOverwrite() throws Exception {
+String tableBasePath = dfsBasePath + "/insert_overwrite";
+// Initial insert
+HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath,
WriteOperationType.INSERT);
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// No new data => no commits.
+cfg.sourceLimit = 0;
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// insert overwrite
+cfg.sourceLimit = 1000;
+cfg.operation = WriteOperationType.INSERT_OVERWRITE;
Review comment:
hi @nsivabalan ,
for the 2nd point
do we need to cover the test case here, because I see test
(https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientOnCopyOnWriteStorage.java#L1227)
This test case is to check Deltastreamer when `op` is `INSERT_OVERWRITE`.
Moreover using the same `SparkRDDWriteClient`. I feel it would be redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add INSERT_OVERWRITE support to DeltaStreamer
> -
>
> Key: HUDI-1860
> URL: https://issues.apache.org/jira/browse/HUDI-1860
> Project: Apache Hudi
> Issue Type: Sub-task
>Reporter: Sagar Sumit
>Assignee: Samrat Deb
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> As discussed in [this
> RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller],
> having full fetch mode use the inser_overwrite to write to sync would be
> better as it can handle schema changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] Samrat002 commented on a change in pull request #3184: [HUDI-1860] Add INSERT_OVERWRITE and INSERT_OVERWRITE_TABLE support to DeltaStreamer
Samrat002 commented on a change in pull request #3184:
URL: https://github.com/apache/hudi/pull/3184#discussion_r671598590
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -1695,6 +1695,54 @@ public void
testJdbcSourceIncrementalFetchInContinuousMode() {
}
}
+ @Test
+ public void testInsertOverwrite() throws Exception {
+String tableBasePath = dfsBasePath + "/insert_overwrite";
+// Initial insert
+HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath,
WriteOperationType.INSERT);
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// No new data => no commits.
+cfg.sourceLimit = 0;
+new HoodieDeltaStreamer(cfg, jsc).sync();
+TestHelpers.assertRecordCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertDistanceCount(1000, tableBasePath + "/*/*.parquet",
sqlContext);
+TestHelpers.assertCommitMetadata("0", tableBasePath, dfs, 1);
+// insert overwrite
+cfg.sourceLimit = 1000;
+cfg.operation = WriteOperationType.INSERT_OVERWRITE;
Review comment:
hi @nsivabalan ,
for the 2nd point
do we need to cover the test case here, because I see test
(https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientOnCopyOnWriteStorage.java#L1227)
This test case is to check Deltastreamer when `op` is `INSERT_OVERWRITE`.
Moreover using the same `SparkRDDWriteClient`. I feel it would be redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (HUDI-2187) Hive integration Improvment
zhangminglei created HUDI-2187: -- Summary: Hive integration Improvment Key: HUDI-2187 URL: https://issues.apache.org/jira/browse/HUDI-2187 Project: Apache Hudi Issue Type: Improvement Components: Hive Integration Reporter: zhangminglei Assignee: zhangminglei See the details from RFC doc https://cwiki.apache.org/confluence/display/HUDI/RFC+-+31%3A+Hive+integration+Improvment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382413#comment-17382413 ] ASF GitHub Bot commented on HUDI-2137: -- hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * 3822f58217e171cbf3ad513b4700415496520dff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=924) * ad8dc0793252287cd160e9bca9cc678f172afd2d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=977) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make hudi examples runnable locally > --- > > Key: HUDI-2137 > URL: https://issues.apache.org/jira/browse/HUDI-2137 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * 3822f58217e171cbf3ad513b4700415496520dff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=924) * ad8dc0793252287cd160e9bca9cc678f172afd2d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=977) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2137) Make hudi examples runnable locally
[ https://issues.apache.org/jira/browse/HUDI-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382411#comment-17382411 ] ASF GitHub Bot commented on HUDI-2137: -- hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * 3822f58217e171cbf3ad513b4700415496520dff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=924) * ad8dc0793252287cd160e9bca9cc678f172afd2d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make hudi examples runnable locally > --- > > Key: HUDI-2137 > URL: https://issues.apache.org/jira/browse/HUDI-2137 > Project: Apache Hudi > Issue Type: Bug >Reporter: Xianghu Wang >Assignee: Xianghu Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3244: [HUDI-2137][WIP] Make hudi examples runnable locally
hudi-bot edited a comment on pull request #3244: URL: https://github.com/apache/hudi/pull/3244#issuecomment-876387479 ## CI report: * 3822f58217e171cbf3ad513b4700415496520dff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=924) * ad8dc0793252287cd160e9bca9cc678f172afd2d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2161) Add support to disable meta column to BulkInsert Row Writer path
[ https://issues.apache.org/jira/browse/HUDI-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382405#comment-17382405 ] ASF GitHub Bot commented on HUDI-2161: -- hudi-bot edited a comment on pull request #3247: URL: https://github.com/apache/hudi/pull/3247#issuecomment-876918931 ## CI report: * e32037e79596e3bf19415d1af107850643ee9ee5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=976) * 0f4199f559dc2aa205ea7109a5b6c0c7e4d34271 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add support to disable meta column to BulkInsert Row Writer path > > > Key: HUDI-2161 > URL: https://issues.apache.org/jira/browse/HUDI-2161 > Project: Apache Hudi > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Objective here is to disable all meta columns so as to avoid storage cost. > Also, some benefits could be seen in write latency with row writer path as no > special handling is required at RowCreateHandle layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
