[GitHub] [hudi] hudi-bot commented on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot commented on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1070350319 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * d8d62a5172a872f65aaca78ee5b3e8bc9c5d3648 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7019) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot removed a comment on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1070319717 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * 438eecf51f0a2c08270ff6c1bca0ea2e4e83c594 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7014) * d8d62a5172a872f65aaca78ee5b3e8bc9c5d3648 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7019) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated (95e6e53 -> 5ba2d9a)
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 95e6e53 [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode (#4975) add 5ba2d9a [HUDI-3494] Consider triggering condition of MOR compaction during archival (#4974) No new revisions were added by this update. Summary of changes: .../apache/hudi/client/HoodieTimelineArchiver.java | 39 - .../compact/ScheduleCompactionActionExecutor.java | 29 ++-- .../apache/hudi/io/TestHoodieTimelineArchiver.java | 167 - .../apache/hudi/common/util/CompactionUtils.java | 83 +- .../hudi/common/util/TestCompactionUtils.java | 120 ++- 5 files changed, 400 insertions(+), 38 deletions(-)
[GitHub] [hudi] nsivabalan merged pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival
nsivabalan merged pull request #4974: URL: https://github.com/apache/hudi/pull/4974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3404) Auto adjust configs for metadata table enablement or turn off metadata table if necessary
[ https://issues.apache.org/jira/browse/HUDI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3404: - Issue Type: Improvement (was: Task) > Auto adjust configs for metadata table enablement or turn off metadata table > if necessary > - > > Key: HUDI-3404 > URL: https://issues.apache.org/jira/browse/HUDI-3404 > Project: Apache Hudi > Issue Type: Improvement > Components: writer-core >Affects Versions: 0.11.0 >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Blocker > Labels: pull-request-available > Fix For: 0.11.0 > > > This is to raise an error in cases where its not safe to use metadata table. > e.g without configuring a lock provider. > For single writer with async table services, the configs can be auto-adjusted > to enable metadata table by default based on the concurrency control > guideline: > https://hudi.apache.org/docs/next/concurrency_control/#enabling-multi-writing -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (HUDI-3404) Auto adjust configs for metadata table enablement or turn off metadata table if necessary
[ https://issues.apache.org/jira/browse/HUDI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-3404. Resolution: Done > Auto adjust configs for metadata table enablement or turn off metadata table > if necessary > - > > Key: HUDI-3404 > URL: https://issues.apache.org/jira/browse/HUDI-3404 > Project: Apache Hudi > Issue Type: Task > Components: writer-core >Affects Versions: 0.11.0 >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Blocker > Labels: pull-request-available > Fix For: 0.11.0 > > > This is to raise an error in cases where its not safe to use metadata table. > e.g without configuring a lock provider. > For single writer with async table services, the configs can be auto-adjusted > to enable metadata table by default based on the concurrency control > guideline: > https://hudi.apache.org/docs/next/concurrency_control/#enabling-multi-writing -- This message was sent by Atlassian Jira (v8.20.1#820001)
[hudi] branch master updated (8ca9a54 -> 95e6e53)
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 8ca9a54 [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator (#4994) add 95e6e53 [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode (#4975) No new revisions were added by this update. Summary of changes: .../org/apache/hudi/config/HoodieWriteConfig.java | 68 - .../apache/hudi/config/TestHoodieWriteConfig.java | 280 - 2 files changed, 272 insertions(+), 76 deletions(-)
[GitHub] [hudi] nsivabalan merged pull request #4975: [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode
nsivabalan merged pull request #4975: URL: https://github.com/apache/hudi/pull/4975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4975: [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode
nsivabalan commented on pull request #4975: URL: https://github.com/apache/hudi/pull/4975#issuecomment-1070339481 Good job on the patch Ethan! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070243213 ## CI report: * 69b821217672dfff8fe3a3b058a480579e8f5e68 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7017) * 5157bbc772c7019ed3f8baaef9618e7d516fde75 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7018) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070320957 ## CI report: * 5157bbc772c7019ed3f8baaef9618e7d516fde75 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7018) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot removed a comment on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1070311808 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * 438eecf51f0a2c08270ff6c1bca0ea2e4e83c594 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7014) * d8d62a5172a872f65aaca78ee5b3e8bc9c5d3648 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot commented on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1070319717 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * 438eecf51f0a2c08270ff6c1bca0ea2e4e83c594 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7014) * d8d62a5172a872f65aaca78ee5b3e8bc9c5d3648 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7019) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot commented on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1070311808 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * 438eecf51f0a2c08270ff6c1bca0ea2e4e83c594 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7014) * d8d62a5172a872f65aaca78ee5b3e8bc9c5d3648 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot removed a comment on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1069910671 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * 438eecf51f0a2c08270ff6c1bca0ea2e4e83c594 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7014) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070236168 ## CI report: * f6e1db77b0b50dcc049c4f051d1bdd62f2942b9a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7016) * 69b821217672dfff8fe3a3b058a480579e8f5e68 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7017) * 5157bbc772c7019ed3f8baaef9618e7d516fde75 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070243213 ## CI report: * 69b821217672dfff8fe3a3b058a480579e8f5e68 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7017) * 5157bbc772c7019ed3f8baaef9618e7d516fde75 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7018) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070198397 ## CI report: * f6e1db77b0b50dcc049c4f051d1bdd62f2942b9a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7016) * 69b821217672dfff8fe3a3b058a480579e8f5e68 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7017) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070236168 ## CI report: * f6e1db77b0b50dcc049c4f051d1bdd62f2942b9a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7016) * 69b821217672dfff8fe3a3b058a480579e8f5e68 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7017) * 5157bbc772c7019ed3f8baaef9618e7d516fde75 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070198397 ## CI report: * f6e1db77b0b50dcc049c4f051d1bdd62f2942b9a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7016) * 69b821217672dfff8fe3a3b058a480579e8f5e68 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7017) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070181104 ## CI report: * d4b2b9aa43813e845c10d3284d600caa5a86b819 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7013) * f6e1db77b0b50dcc049c4f051d1bdd62f2942b9a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7016) * 69b821217672dfff8fe3a3b058a480579e8f5e68 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070014635 ## CI report: * d4b2b9aa43813e845c10d3284d600caa5a86b819 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7013) * f6e1db77b0b50dcc049c4f051d1bdd62f2942b9a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7016) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070181104 ## CI report: * d4b2b9aa43813e845c10d3284d600caa5a86b819 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7013) * f6e1db77b0b50dcc049c4f051d1bdd62f2942b9a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7016) * 69b821217672dfff8fe3a3b058a480579e8f5e68 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix
alexeykudinkin commented on pull request #5052: URL: https://github.com/apache/hudi/pull/5052#issuecomment-1070091573 @sekaiga can you please fill out the description properly? Is this PR still WIP? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HUDI-3020) Create a utility to generate mainfest file
[ https://issues.apache.org/jira/browse/HUDI-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joyan Sil reassigned HUDI-3020: --- Assignee: Joyan Sil (was: Prashant Wason) > Create a utility to generate mainfest file > -- > > Key: HUDI-3020 > URL: https://issues.apache.org/jira/browse/HUDI-3020 > Project: Apache Hudi > Issue Type: Task >Reporter: Vinoth Govindarajan >Assignee: Joyan Sil >Priority: Major > > Create a utility to generate manifest file which contains the latest snapshot > files for each partition in a CSV format with only one column filename. > > This is the first step towards integrating hudi with snowflake. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] danny0405 commented on a change in pull request #3304: [HUDI-2198] Clean and reset the bootstrap events for coordinator when…
danny0405 commented on a change in pull request #3304:
URL: https://github.com/apache/hudi/pull/3304#discussion_r828713714
##
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/BucketAssignFunction.java
##
@@ -194,9 +194,10 @@ private void processRecord(HoodieRecord record,
Collector out) throws Exce
} else {
location = getNewRecordLocation(partitionPath);
this.context.setCurrentKey(recordKey);
- if (isChangingRecords) {
-updateIndexState(partitionPath, location);
- }
+}
+// always refresh the index
+if (isChangingRecords) {
+ updateIndexState(partitionPath, location);
Review comment:
No, but the code can be promoted indeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069998952 ## CI report: * d4b2b9aa43813e845c10d3284d600caa5a86b819 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7013) * f6e1db77b0b50dcc049c4f051d1bdd62f2942b9a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1070014635 ## CI report: * d4b2b9aa43813e845c10d3284d600caa5a86b819 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7013) * f6e1db77b0b50dcc049c4f051d1bdd62f2942b9a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7016) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069998952 ## CI report: * d4b2b9aa43813e845c10d3284d600caa5a86b819 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7013) * f6e1db77b0b50dcc049c4f051d1bdd62f2942b9a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][Stacked on 4996][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069803516 ## CI report: * d4b2b9aa43813e845c10d3284d600caa5a86b819 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7013) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot removed a comment on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1069800698 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * a19a778f36d2ad00357691d4a04a0da14f9b624f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7003) * 438eecf51f0a2c08270ff6c1bca0ea2e4e83c594 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7014) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot commented on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1069910671 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * 438eecf51f0a2c08270ff6c1bca0ea2e4e83c594 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7014) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] tjtoll edited a comment on issue #5047: [SUPPORT] Small file creation while writing to a Hudi Table
tjtoll edited a comment on issue #5047: URL: https://github.com/apache/hudi/issues/5047#issuecomment-1069788126 I am experiencing the same exact problem - also with AWS Glue. I tried the below settings and they increased the file sizes to a more appropriate size, but the insert is taking 10x longer. Also interested in some guidance on this issue. 'hoodie.copyonwrite.insert.auto.split': 'false', 'hoodie.copyonwrite.insert.split.size': 1, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3643) Hive count throws exception when the table is empty and the path depth is less than 3
[
https://issues.apache.org/jira/browse/HUDI-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
董可伦 updated HUDI-3643:
--
Description:
Spark SQL create non-partition hudi table:
{code:java}
create table test_hudi_table (
id int,
name string,
price double,
ts long,
dt string
) using hudi
options (
primaryKey = 'id',
preCombineField = 'ts',
type = 'cow'
)
location '/tmp/test_hudi_table';{code}
hive tez count
{code:java}
select count(1) from test_hudi_table;{code}
then exception:
hudi 0.9.0
{code:java}
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1,
vertexId=vertex_1647336877182_0100_4_00, diagnostics=[Vertex
vertex_1647336877182_0100_4_00 [Map 1] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: test_hudi_table initializer failed,
vertex=vertex_1647336877182_0100_4_00 [Map 1], java.lang.NullPointerException
at
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils.getTableMetaClientForBasePath(HoodieInputFormatUtils.java:327)
at
org.apache.hudi.hadoop.InputPathHandler.parseInputPaths(InputPathHandler.java:107)
at
org.apache.hudi.hadoop.InputPathHandler.(InputPathHandler.java:68)
at
org.apache.hudi.hadoop.HoodieParquetInputFormat.listStatus(HoodieParquetInputFormat.java:80)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
hudi master also exception
{code:java}
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1,
vertexId=vertex_1647336877182_0106_1_00, diagnostics=[Vertex
vertex_1647336877182_0106_1_00 [Map 1] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: test_hudi_table initializer failed,
vertex=vertex_1647336877182_0106_1_00 [Map 1], java.lang.NullPointerException
at
org.apache.hudi.hadoop.utils.HoodieInputFormatUtils.getTableMetaClientForBasePathUnchecked(HoodieInputFormatUtils.java:335)
at
org.apache.hudi.hadoop.InputPathHandler.parseInputPaths(InputPathHandler.java:110)
at
org.apache.hudi.hadoop.InputPathHandler.(InputPathHandler.java:72)
at
org.apache.hudi.hadoop.HoodieCopyOnWriteTableInputFormat.listStatus(HoodieCopyOnWriteTableInputFormat.java:109)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325)
at
org.apache.hudi.hadoop.HoodieParquetInputFormatBase.getSplits(HoodieParquetInputFormatBase.java:68)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
[jira] [Updated] (HUDI-3643) Hive count throws exception when the table is empty and the path depth is less than 3
[ https://issues.apache.org/jira/browse/HUDI-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 董可伦 updated HUDI-3643: -- Description: Spark SQL create non-partition hudi table: ```sql create table test_hudi_table ( id int, name string, price double, ts long, dt string ) using hudi options ( primaryKey = 'id', preCombineField = 'ts', type = 'cow' ) location '/tmp/test_hudi_table'; ``` hive tez count ```sql select count(1) from test_hudi_table; ``` hen exception: hudi 0.9.0 ```java ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1647336877182_0100_4_00, diagnostics=[Vertex vertex_1647336877182_0100_4_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: test_hudi_table initializer failed, vertex=vertex_1647336877182_0100_4_00 [Map 1], java.lang.NullPointerException at org.apache.hudi.hadoop.utils.HoodieInputFormatUtils.getTableMetaClientForBasePath(HoodieInputFormatUtils.java:327) at org.apache.hudi.hadoop.InputPathHandler.parseInputPaths(InputPathHandler.java:107) at org.apache.hudi.hadoop.InputPathHandler.(InputPathHandler.java:68) at org.apache.hudi.hadoop.HoodieParquetInputFormat.listStatus(HoodieParquetInputFormat.java:80) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325) at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779) at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` hudi master also exception ```java ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1647336877182_0106_1_00, diagnostics=[Vertex vertex_1647336877182_0106_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: test_hudi_table initializer failed, vertex=vertex_1647336877182_0106_1_00 [Map 1], java.lang.NullPointerException at org.apache.hudi.hadoop.utils.HoodieInputFormatUtils.getTableMetaClientForBasePathUnchecked(HoodieInputFormatUtils.java:335) at org.apache.hudi.hadoop.InputPathHandler.parseInputPaths(InputPathHandler.java:110) at org.apache.hudi.hadoop.InputPathHandler.(InputPathHandler.java:72) at org.apache.hudi.hadoop.HoodieCopyOnWriteTableInputFormat.listStatus(HoodieCopyOnWriteTableInputFormat.java:109) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325) at org.apache.hudi.hadoop.HoodieParquetInputFormatBase.getSplits(HoodieParquetInputFormatBase.java:68) at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779) at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at
[jira] [Updated] (HUDI-3643) Hive count throws exception when the table is empty and the path depth is less than 3
[ https://issues.apache.org/jira/browse/HUDI-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 董可伦 updated HUDI-3643: -- Description: ## Spark SQL create non-partition hudi table: ```sql create table test_hudi_table ( id int, name string, price double, ts long, dt string ) using hudi options ( primaryKey = 'id', preCombineField = 'ts', type = 'cow' ) location '/tmp/test_hudi_table'; ``` ## hive tez count ```sql select count(1) from test_hudi_table; ``` ## then exception: ### hudi 0.9.0 ```java ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1647336877182_0100_4_00, diagnostics=[Vertex vertex_1647336877182_0100_4_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: test_hudi_table initializer failed, vertex=vertex_1647336877182_0100_4_00 [Map 1], java.lang.NullPointerException at org.apache.hudi.hadoop.utils.HoodieInputFormatUtils.getTableMetaClientForBasePath(HoodieInputFormatUtils.java:327) at org.apache.hudi.hadoop.InputPathHandler.parseInputPaths(InputPathHandler.java:107) at org.apache.hudi.hadoop.InputPathHandler.(InputPathHandler.java:68) at org.apache.hudi.hadoop.HoodieParquetInputFormat.listStatus(HoodieParquetInputFormat.java:80) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325) at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779) at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` ### hudi master also exception ```java ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1647336877182_0106_1_00, diagnostics=[Vertex vertex_1647336877182_0106_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: test_hudi_table initializer failed, vertex=vertex_1647336877182_0106_1_00 [Map 1], java.lang.NullPointerException at org.apache.hudi.hadoop.utils.HoodieInputFormatUtils.getTableMetaClientForBasePathUnchecked(HoodieInputFormatUtils.java:335) at org.apache.hudi.hadoop.InputPathHandler.parseInputPaths(InputPathHandler.java:110) at org.apache.hudi.hadoop.InputPathHandler.(InputPathHandler.java:72) at org.apache.hudi.hadoop.HoodieCopyOnWriteTableInputFormat.listStatus(HoodieCopyOnWriteTableInputFormat.java:109) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325) at org.apache.hudi.hadoop.HoodieParquetInputFormatBase.getSplits(HoodieParquetInputFormatBase.java:68) at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:779) at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at
[GitHub] [hudi] hudi-bot commented on pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival
hudi-bot commented on pull request #4974: URL: https://github.com/apache/hudi/pull/4974#issuecomment-1069822816 ## CI report: * 681207d07b3297c25fa32a02d6570dc9110c8aa7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7010) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival
hudi-bot removed a comment on pull request #4974: URL: https://github.com/apache/hudi/pull/4974#issuecomment-1069779096 ## CI report: * 8f7ed79ebab7d10624e7e5e9a641cc41e802ef98 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6659) * 681207d07b3297c25fa32a02d6570dc9110c8aa7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7010) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] prashantwason commented on pull request #4274: [HUDI-2974] Make the prefix for metrics name configurable
prashantwason commented on pull request #4274:
URL: https://github.com/apache/hudi/pull/4274#issuecomment-1069818077
@yihua @nsivabalan
I think this commit broke metrics for usage like this:
HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
.withExecutorMetrics(true).on(true);
HoodieWriteConfig writeConfig =
HoodieWriteConfig.newBuilder().withPath("/tmp").forTable(tableName)
.withMetricsConfig(metricsConfig.build())
.withMetricsGraphiteConfig(HoodieMetricsGraphiteConfig.newBuilder().usePrefix(metricPrefix).build())
.build();
I end up with commonMetricPrefix as "" in my runs.
Also I do not see any setter for METRICS_REPORTER_PREFIX in
HoodieMetricsConfig.Builder.
How are we to use this correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4962: [HUDI-3355] Issue with out of order commits in the timeline when ingestion writers using SparkAllowUpdateStrategy
yihua commented on a change in pull request #4962: URL: https://github.com/apache/hudi/pull/4962#discussion_r828623933 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java ## @@ -120,6 +121,7 @@ protected transient AsyncArchiveService asyncArchiveService; protected final TransactionManager txnManager; protected Option>> lastCompletedTxnAndMetadata = Option.empty(); + protected List pendingReplaceRequestedInstants = new ArrayList<>(); Review comment: +1. This can happen for multi-writer and echoes my comment above. The c2 is from backfill and takes a long time to finish, that can cause the same issue as the replacecommit case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated (91849c3 -> 8ca9a54)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 91849c3 [HUDI-3607] Support backend switch in HoodieFlinkStreamer (#5032) add 8ca9a54 [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator (#4994) No new revisions were added by this update. Summary of changes: .../utilities/HoodieMetadataTableValidator.java| 176 + 1 file changed, 146 insertions(+), 30 deletions(-)
[GitHub] [hudi] yihua merged pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
yihua merged pull request #4994: URL: https://github.com/apache/hudi/pull/4994 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4975: [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode
hudi-bot removed a comment on pull request #4975: URL: https://github.com/apache/hudi/pull/4975#issuecomment-1069770565 ## CI report: * 0c1aad3b608f4eea6d5882b27e819ee138dd3748 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6673) * 1bc5f116d28f5ef2f6ea522a71688ea49c04b693 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7009) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4975: [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode
hudi-bot commented on pull request #4975: URL: https://github.com/apache/hudi/pull/4975#issuecomment-1069813192 ## CI report: * 1bc5f116d28f5ef2f6ea522a71688ea49c04b693 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7009) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] kywe665 edited a comment on pull request #5055: [MINOR] - blog css adj and adding images
kywe665 edited a comment on pull request #5055: URL: https://github.com/apache/hudi/pull/5055#issuecomment-1069810983 I will do pages 2 & 3 soon, but wanted to push this one out first. Next steps: 1. I will add author and source 2. I will merge talks and articles in 3. I will apply images to the other hudi blogs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] kywe665 commented on pull request #5055: [MINOR] - blog css adj and adding images
kywe665 commented on pull request #5055: URL: https://github.com/apache/hudi/pull/5055#issuecomment-1069810983 I will do pages 2 & 3 soon, but wanted to push this one out first. Next I will add Author and Source -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4994: [Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator
yihua commented on a change in pull request #4994:
URL: https://github.com/apache/hudi/pull/4994#discussion_r828610748
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieMetadataTableValidator.java
##
@@ -345,16 +358,59 @@ public void doMetadataTableValidation() {
boolean finalResult = true;
metaClient.reloadActiveTimeline();
String basePath = metaClient.getBasePath();
+List baseFilesUnderDeletion = Collections.emptyList();
+
+if (cfg.skipUnderDeletionDataFiles) {
+ HoodieTimeline pendingCleaningTimeline = metaClient.getActiveTimeline()
+ .getCleanerTimeline()
+ .filter(instant -> instant.getState() !=
HoodieInstant.State.COMPLETED);
Review comment:
Sg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] kywe665 commented on pull request #5055: [MINOR] - blog css adj and adding images
kywe665 commented on pull request #5055: URL: https://github.com/apache/hudi/pull/5055#issuecomment-1069810122  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069803516 ## CI report: * d4b2b9aa43813e845c10d3284d600caa5a86b819 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7013) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069800677 ## CI report: * 846b0d8949a8178d5d203970fa4baffb5a1163d8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7012) * d4b2b9aa43813e845c10d3284d600caa5a86b819 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7013) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3650) Revisit all usages of filterPendingCompactionTimeline()
[ https://issues.apache.org/jira/browse/HUDI-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3650: Fix Version/s: 0.11.0 > Revisit all usages of filterPendingCompactionTimeline() > > > Key: HUDI-3650 > URL: https://issues.apache.org/jira/browse/HUDI-3650 > Project: Apache Hudi > Issue Type: Bug >Reporter: Ethan Guo >Priority: Critical > Fix For: 0.11.0 > > > [https://github.com/apache/hudi/pull/4172/files] > > We need to find all usages of filterPendingCompactionTimeline and replace > them with new methods. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3650) Revisit all usages of filterPendingCompactionTimeline()
[ https://issues.apache.org/jira/browse/HUDI-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3650: Affects Version/s: (was: 0.11.0) > Revisit all usages of filterPendingCompactionTimeline() > > > Key: HUDI-3650 > URL: https://issues.apache.org/jira/browse/HUDI-3650 > Project: Apache Hudi > Issue Type: Bug >Reporter: Ethan Guo >Priority: Critical > > [https://github.com/apache/hudi/pull/4172/files] > > We need to find all usages of filterPendingCompactionTimeline and replace > them with new methods. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] yihua commented on a change in pull request #4172: [HUDI-2892][BUG]Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results
yihua commented on a change in pull request #4172: URL: https://github.com/apache/hudi/pull/4172#discussion_r828592002 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java ## @@ -158,8 +158,12 @@ public boolean archiveIfRequired(HoodieEngineContext context) throws IOException // TODO (na) : Add a way to return actions associated with a timeline and then merge/unify // with logic above to avoid Stream.concats HoodieTimeline commitTimeline = table.getCompletedCommitsTimeline(); -Option oldestPendingCompactionInstant = - table.getActiveTimeline().filterPendingCompactionTimeline().firstInstant(); + +Option oldestPendingCompactionAndReplaceInstant = table.getActiveTimeline() + .getTimelineOfActions(CollectionUtils.createSet(HoodieTimeline.COMPACTION_ACTION, HoodieTimeline.REPLACE_COMMIT_ACTION)) Review comment: Jira ticket created: https://issues.apache.org/jira/browse/HUDI-3650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-3650) Revisit all usages of filterPendingCompactionTimeline()
Ethan Guo created HUDI-3650: --- Summary: Revisit all usages of filterPendingCompactionTimeline() Key: HUDI-3650 URL: https://issues.apache.org/jira/browse/HUDI-3650 Project: Apache Hudi Issue Type: Bug Affects Versions: 0.11.0 Reporter: Ethan Guo [https://github.com/apache/hudi/pull/4172/files] We need to find all usages of filterPendingCompactionTimeline and replace them with new methods. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069800677 ## CI report: * 846b0d8949a8178d5d203970fa4baffb5a1163d8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7012) * d4b2b9aa43813e845c10d3284d600caa5a86b819 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7013) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069799054 ## CI report: * 088c5bbb80c355ff88d72b8efa5618c4d3118dee Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7011) * 846b0d8949a8178d5d203970fa4baffb5a1163d8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7012) * d4b2b9aa43813e845c10d3284d600caa5a86b819 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot removed a comment on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1069799095 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * a19a778f36d2ad00357691d4a04a0da14f9b624f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7003) * 438eecf51f0a2c08270ff6c1bca0ea2e4e83c594 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot commented on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1069800698 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * a19a778f36d2ad00357691d4a04a0da14f9b624f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7003) * 438eecf51f0a2c08270ff6c1bca0ea2e4e83c594 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7014) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot commented on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1069799095 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * a19a778f36d2ad00357691d4a04a0da14f9b624f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7003) * 438eecf51f0a2c08270ff6c1bca0ea2e4e83c594 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4957: [HUDI-3406] Rollback incorrectly relying on FS listing instead of Com…
hudi-bot removed a comment on pull request #4957: URL: https://github.com/apache/hudi/pull/4957#issuecomment-1069322997 ## CI report: * 9ba5c351c32f9f364c30b4bc9a814075150d9728 UNKNOWN * 53b4cec7d05dd0bb5a622ffb33597471da7711de UNKNOWN * a19a778f36d2ad00357691d4a04a0da14f9b624f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7003) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069796629 ## CI report: * ec422719c01c7d5d4418276f18e7be3022faac5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6563) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6574) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6661) * 088c5bbb80c355ff88d72b8efa5618c4d3118dee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7011) * 846b0d8949a8178d5d203970fa4baffb5a1163d8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069799054 ## CI report: * 088c5bbb80c355ff88d72b8efa5618c4d3118dee Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7011) * 846b0d8949a8178d5d203970fa4baffb5a1163d8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7012) * d4b2b9aa43813e845c10d3284d600caa5a86b819 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069796629 ## CI report: * ec422719c01c7d5d4418276f18e7be3022faac5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6563) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6574) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6661) * 088c5bbb80c355ff88d72b8efa5618c4d3118dee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7011) * 846b0d8949a8178d5d203970fa4baffb5a1163d8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069794614 ## CI report: * ec422719c01c7d5d4418276f18e7be3022faac5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6563) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6574) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6661) * 088c5bbb80c355ff88d72b8efa5618c4d3118dee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7011) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069792521 ## CI report: * ec422719c01c7d5d4418276f18e7be3022faac5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6563) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6574) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6661) * 088c5bbb80c355ff88d72b8efa5618c4d3118dee UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069794614 ## CI report: * ec422719c01c7d5d4418276f18e7be3022faac5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6563) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6574) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6661) * 088c5bbb80c355ff88d72b8efa5618c4d3118dee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7011) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot removed a comment on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1061216142 ## CI report: * ec422719c01c7d5d4418276f18e7be3022faac5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6563) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6574) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6661) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4955: [HUDI-3549][WIP][DO_NOT_MERGE] Removing "spark-avro" bundling from Hudi by default
hudi-bot commented on pull request #4955: URL: https://github.com/apache/hudi/pull/4955#issuecomment-1069792521 ## CI report: * ec422719c01c7d5d4418276f18e7be3022faac5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6563) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6574) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6661) * 088c5bbb80c355ff88d72b8efa5618c4d3118dee UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] tjtoll commented on issue #5047: [SUPPORT] Small file creation while writing to a Hudi Table
tjtoll commented on issue #5047: URL: https://github.com/apache/hudi/issues/5047#issuecomment-1069788126 I am experiencing the same exact problem. I tried the below settings and they increased the file sizes to a more appropriate size, but the insert is taking 10x longer. Also interested in some guidance on this issue. 'hoodie.copyonwrite.insert.auto.split': 'false', 'hoodie.copyonwrite.insert.split.size': 1, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4856: [HUDI-2439] Replace RDD with HoodieData in HoodieSparkTable and commit executors
yihua commented on a change in pull request #4856:
URL: https://github.com/apache/hudi/pull/4856#discussion_r828562205
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/commit/TestDeleteHelper.java
##
@@ -1,194 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.action.commit;
-
-import org.apache.hudi.client.WriteStatus;
-import org.apache.hudi.client.common.HoodieSparkEngineContext;
-import org.apache.hudi.common.model.EmptyHoodieRecordPayload;
-import org.apache.hudi.common.model.HoodieKey;
-import org.apache.hudi.common.model.HoodieRecord;
-import org.apache.hudi.config.HoodieWriteConfig;
-import org.apache.hudi.data.HoodieJavaRDD;
-import org.apache.hudi.index.HoodieIndex;
-import org.apache.hudi.table.HoodieTable;
-import org.apache.hudi.table.action.HoodieWriteMetadata;
-
-import org.apache.spark.Partition;
-import org.apache.spark.api.java.JavaPairRDD;
-import org.apache.spark.api.java.JavaRDD;
-import org.apache.spark.api.java.JavaSparkContext;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-import org.junit.jupiter.api.extension.ExtendWith;
-import org.mockito.Mock;
-import org.mockito.junit.jupiter.MockitoExtension;
-
-import java.util.Collections;
-import java.util.List;
-
-import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyInt;
-import static org.mockito.ArgumentMatchers.anyString;
-import static org.mockito.ArgumentMatchers.eq;
-import static org.mockito.Mockito.doNothing;
-import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.never;
-import static org.mockito.Mockito.times;
-import static org.mockito.Mockito.verify;
-import static org.mockito.Mockito.when;
-
-@ExtendWith(MockitoExtension.class)
-public class TestDeleteHelper {
Review comment:
Sg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] tisonkun commented on a change in pull request #3304: [HUDI-2198] Clean and reset the bootstrap events for coordinator when…
tisonkun commented on a change in pull request #3304:
URL: https://github.com/apache/hudi/pull/3304#discussion_r828561997
##
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/BucketAssignFunction.java
##
@@ -194,9 +194,10 @@ private void processRecord(HoodieRecord record,
Collector out) throws Exce
} else {
location = getNewRecordLocation(partitionPath);
this.context.setCurrentKey(recordKey);
- if (isChangingRecords) {
-updateIndexState(partitionPath, location);
- }
+}
+// always refresh the index
+if (isChangingRecords) {
+ updateIndexState(partitionPath, location);
Review comment:
After this change, shall L189 be redundant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4856: [HUDI-2439] Replace RDD with HoodieData in HoodieSparkTable and commit executors
yihua commented on a change in pull request #4856: URL: https://github.com/apache/hudi/pull/4856#discussion_r828561373 ## File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java ## @@ -363,7 +365,7 @@ public void completeCompaction( // Do not do any conflict resolution here as we do with regular writes. We take the lock here to ensure all writes to metadata table happens within a // single lock (single writer). Because more than one write to metadata table will result in conflicts since all of them updates the same partition. table.getMetadataWriter(compactionInstant.getTimestamp()).ifPresent( - w -> w.update(metadata, compactionInstant.getTimestamp(), table.isTableServiceAction(compactionInstant.getAction(; + w -> ((HoodieTableMetadataWriter) w).update(metadata, compactionInstant.getTimestamp(), table.isTableServiceAction(compactionInstant.getAction(; Review comment: Per discussion, this is needed due to removal of generics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] kywe665 opened a new pull request #5055: [MINOR] - blog css adj and adding images
kywe665 opened a new pull request #5055: URL: https://github.com/apache/hudi/pull/5055 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose of the pull request blog page improvements ## Brief change log - added thumbnail images for all first page blogs. - adjusted css so it stays same size regardless of titles and image sizes ## Verify this pull request docs change only ## Committer checklist - [X] Has a corresponding JIRA in PR title & commit - [X] Commit message is descriptive of the change - [X] CI is green - [X] Necessary doc changes done or have another open PR - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3610) Validate Hudi Kafka Connect Sink writing to S3
[
https://issues.apache.org/jira/browse/HUDI-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-3610:
-
Sprint: Cont' improve - 2022/03/7
> Validate Hudi Kafka Connect Sink writing to S3
> --
>
> Key: HUDI-3610
> URL: https://issues.apache.org/jira/browse/HUDI-3610
> Project: Apache Hudi
> Issue Type: Task
> Components: kafka-connect
>Reporter: Ethan Guo
>Assignee: Raymond Xu
>Priority: Critical
> Fix For: 0.11.0
>
>
> From community:
> Hi guys, I'm trying to implement this architecture with hudi
> db table — Debezium --> kafka ---Hudi sink connector --> S3 bucket
> My setting
> Kafka version 2.4
> Hudi version 0.10.1
> Hdf sink connector version 10.1.4
> I'm encountering this error
> {code:java}
> ERROR WorkerSinkTask{id=} Task threw an uncaught and unrecoverable
> exception. Task is being killed and will not recover until manually restarted
> (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
> at org.apache.hudi.connect.HoodieSinkTask.start(HoodieSinkTask.java:80)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:312)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:186)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.fs.FSDataInputStream
> at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
> at
> org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:103)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
> ... 9 more {code}
> this is the Dockerfile I used to bake the custom image
> {code:java}
> #==
> FROM maven:3.8.4-openjdk-8-slim as build-hudi
> ENV HUDI_VERSION=0.10.1
> RUN mkdir /home/hudi && \
> curl -L
> https://github.com/apache/hudi/archive/refs/tags/release-$HUDI_VERSION.tar.gz
> \
> > hudi-release-$HUDI_VERSION.tar.gz && \
> tar -xzvf ./hudi-release-$HUDI_VERSION.tar.gz -C /home/hudi && \
> rm ./hudi-release-$HUDI_VERSION.tar.gz && \
> cd /home/hudi/hudi-release-$HUDI_VERSION && \
> mvn package -DskipTests -pl packaging/hudi-kafka-connect-bundle -am
> #==
> FROM confluentinc/cp-kafka-connect:7.0.1
> ENV DEBEZIUM_VERSION=1.4.1.Final \
> MAVEN_REPO_CORE="https://repo1.maven.org/maven2; \
> CONNECTOR=mysql \
> KAFKA_CONNECT_PLUGINS_DIR=/usr/share/java \
> DATAGEN_VERSION=0.5.3 \
> ADX_SINK_CONNECTOR_VERSION=2.2.0 \
> AMAZON_S3_SINK_CONNECTOR_VERSION=10.0.3 \
> HDFS2_SINK_CONNECTOR_VERSION=10.1.4 \
> HUDI_OUTPUT_JAR_FILE="hudi-kafka-connect-bundle-0.11.0-SNAPSHOT.jar" \
> HUDI_VERSION=0.10.1
> RUN curl -fSL -o /tmp/plugin.tar.gz \
>
> $MAVEN_REPO_CORE/io/debezium/debezium-connector-$CONNECTOR/$DEBEZIUM_VERSION/debezium-connector-$CONNECTOR-$DEBEZIUM_VERSION-plugin.tar.gz
> && \
> tar -xzf /tmp/plugin.tar.gz -C $KAFKA_CONNECT_PLUGINS_DIR && \
> rm -f /tmp/plugin.tar.gz
> RUN confluent-hub install --no-prompt
> confluentinc/kafka-connect-datagen:$DATAGEN_VERSION && \
> confluent-hub install --no-prompt
> microsoftcorporation/kafka-sink-azure-kusto:$ADX_SINK_CONNECTOR_VERSION && \
> confluent-hub install --no-prompt
> confluentinc/kafka-connect-s3:$AMAZON_S3_SINK_CONNECTOR_VERSION && \
> confluent-hub install --no-prompt
> confluentinc/kafka-connect-hdfs:$HDFS2_SINK_CONNECTOR_VERSION
> COPY --from=build-hudi
> /home/hudi/hudi-release-$HUDI_VERSION/packaging/hudi-kafka-connect-bundle/target/hudi-kafka-connect-bundle-$HUDI_VERSION.jar
> $KAFKA_CONNECT_PLUGINS_DIR/$HUDI_OUTPUT_JAR_FILE {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HUDI-3610) Validate Hudi Kafka Connect Sink writing to S3
[
https://issues.apache.org/jira/browse/HUDI-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-3610:
-
Component/s: kafka-connect
> Validate Hudi Kafka Connect Sink writing to S3
> --
>
> Key: HUDI-3610
> URL: https://issues.apache.org/jira/browse/HUDI-3610
> Project: Apache Hudi
> Issue Type: Task
> Components: kafka-connect
>Reporter: Ethan Guo
>Assignee: Raymond Xu
>Priority: Critical
> Fix For: 0.11.0
>
>
> From community:
> Hi guys, I'm trying to implement this architecture with hudi
> db table — Debezium --> kafka ---Hudi sink connector --> S3 bucket
> My setting
> Kafka version 2.4
> Hudi version 0.10.1
> Hdf sink connector version 10.1.4
> I'm encountering this error
> {code:java}
> ERROR WorkerSinkTask{id=} Task threw an uncaught and unrecoverable
> exception. Task is being killed and will not recover until manually restarted
> (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
> at org.apache.hudi.connect.HoodieSinkTask.start(HoodieSinkTask.java:80)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:312)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:186)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.fs.FSDataInputStream
> at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
> at
> org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:103)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
> ... 9 more {code}
> this is the Dockerfile I used to bake the custom image
> {code:java}
> #==
> FROM maven:3.8.4-openjdk-8-slim as build-hudi
> ENV HUDI_VERSION=0.10.1
> RUN mkdir /home/hudi && \
> curl -L
> https://github.com/apache/hudi/archive/refs/tags/release-$HUDI_VERSION.tar.gz
> \
> > hudi-release-$HUDI_VERSION.tar.gz && \
> tar -xzvf ./hudi-release-$HUDI_VERSION.tar.gz -C /home/hudi && \
> rm ./hudi-release-$HUDI_VERSION.tar.gz && \
> cd /home/hudi/hudi-release-$HUDI_VERSION && \
> mvn package -DskipTests -pl packaging/hudi-kafka-connect-bundle -am
> #==
> FROM confluentinc/cp-kafka-connect:7.0.1
> ENV DEBEZIUM_VERSION=1.4.1.Final \
> MAVEN_REPO_CORE="https://repo1.maven.org/maven2; \
> CONNECTOR=mysql \
> KAFKA_CONNECT_PLUGINS_DIR=/usr/share/java \
> DATAGEN_VERSION=0.5.3 \
> ADX_SINK_CONNECTOR_VERSION=2.2.0 \
> AMAZON_S3_SINK_CONNECTOR_VERSION=10.0.3 \
> HDFS2_SINK_CONNECTOR_VERSION=10.1.4 \
> HUDI_OUTPUT_JAR_FILE="hudi-kafka-connect-bundle-0.11.0-SNAPSHOT.jar" \
> HUDI_VERSION=0.10.1
> RUN curl -fSL -o /tmp/plugin.tar.gz \
>
> $MAVEN_REPO_CORE/io/debezium/debezium-connector-$CONNECTOR/$DEBEZIUM_VERSION/debezium-connector-$CONNECTOR-$DEBEZIUM_VERSION-plugin.tar.gz
> && \
> tar -xzf /tmp/plugin.tar.gz -C $KAFKA_CONNECT_PLUGINS_DIR && \
> rm -f /tmp/plugin.tar.gz
> RUN confluent-hub install --no-prompt
> confluentinc/kafka-connect-datagen:$DATAGEN_VERSION && \
> confluent-hub install --no-prompt
> microsoftcorporation/kafka-sink-azure-kusto:$ADX_SINK_CONNECTOR_VERSION && \
> confluent-hub install --no-prompt
> confluentinc/kafka-connect-s3:$AMAZON_S3_SINK_CONNECTOR_VERSION && \
> confluent-hub install --no-prompt
> confluentinc/kafka-connect-hdfs:$HDFS2_SINK_CONNECTOR_VERSION
> COPY --from=build-hudi
> /home/hudi/hudi-release-$HUDI_VERSION/packaging/hudi-kafka-connect-bundle/target/hudi-kafka-connect-bundle-$HUDI_VERSION.jar
> $KAFKA_CONNECT_PLUGINS_DIR/$HUDI_OUTPUT_JAR_FILE {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HUDI-3579) Add timeline commands in hudi-cli
[ https://issues.apache.org/jira/browse/HUDI-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3579: - Sprint: Cont' improve - 2022/03/7 > Add timeline commands in hudi-cli > - > > Key: HUDI-3579 > URL: https://issues.apache.org/jira/browse/HUDI-3579 > Project: Apache Hudi > Issue Type: Task > Components: cli >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival
hudi-bot commented on pull request #4974: URL: https://github.com/apache/hudi/pull/4974#issuecomment-1069779096 ## CI report: * 8f7ed79ebab7d10624e7e5e9a641cc41e802ef98 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6659) * 681207d07b3297c25fa32a02d6570dc9110c8aa7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7010) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival
hudi-bot removed a comment on pull request #4974: URL: https://github.com/apache/hudi/pull/4974#issuecomment-1069777232 ## CI report: * 8f7ed79ebab7d10624e7e5e9a641cc41e802ef98 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6659) * 681207d07b3297c25fa32a02d6570dc9110c8aa7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival
hudi-bot commented on pull request #4974: URL: https://github.com/apache/hudi/pull/4974#issuecomment-1069777232 ## CI report: * 8f7ed79ebab7d10624e7e5e9a641cc41e802ef98 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6659) * 681207d07b3297c25fa32a02d6570dc9110c8aa7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival
hudi-bot removed a comment on pull request #4974: URL: https://github.com/apache/hudi/pull/4974#issuecomment-1061374163 ## CI report: * 8f7ed79ebab7d10624e7e5e9a641cc41e802ef98 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6659) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4286: [WIP][HUDI-2955] Upgrade Hadoop to 3.3.1, Hive to 3.1.2, HBase to 2.4.8
hudi-bot commented on pull request #4286: URL: https://github.com/apache/hudi/pull/4286#issuecomment-1069776883 ## CI report: * c2738a5a1cd6f5ea8cffa966d7dc332f2d211f29 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7008) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4286: [WIP][HUDI-2955] Upgrade Hadoop to 3.3.1, Hive to 3.1.2, HBase to 2.4.8
hudi-bot removed a comment on pull request #4286: URL: https://github.com/apache/hudi/pull/4286#issuecomment-1069734507 ## CI report: * dbe231115001abd287e8d948d51011b708c02f7c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6984) * c2738a5a1cd6f5ea8cffa966d7dc332f2d211f29 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7008) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival
yihua commented on a change in pull request #4974:
URL: https://github.com/apache/hudi/pull/4974#discussion_r828554793
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/io/TestHoodieTimelineArchiver.java
##
@@ -946,6 +961,152 @@ public void
testArchiveCommitsWithCompactionCommitInMetadataTableTimeline() thro
}
}
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testArchivalWithMaxDeltaCommitsGuaranteeForCompaction(boolean
enableMetadata) throws Exception {
+HoodieWriteConfig writeConfig = initTestTableAndGetWriteConfig(
+enableMetadata, 2, 4, 8, 1, HoodieTableType.MERGE_ON_READ);
+
+// When max archival commits is set to 4, even after 8 delta commits,
since the number of delta
+// commits is still smaller than 8, the archival should not kick in.
+// The archival should only kick in after the 9th delta commit
+// instant "0001" to "0009"
+for (int i = 1; i < 10; i++) {
+ testTable.doWriteOperation("000" + i, WriteOperationType.UPSERT, i
== 1
+ ? Arrays.asList("p1", "p2") : Collections.emptyList(),
Arrays.asList("p1", "p2"), 2);
+ // archival
+ Pair, List> commitsList =
archiveAndGetCommitsList(writeConfig);
+ List originalCommits = commitsList.getKey();
+ List commitsAfterArchival = commitsList.getValue();
+
+ if (i <= 8) {
+assertEquals(originalCommits, commitsAfterArchival);
+ } else {
+assertEquals(1, originalCommits.size() - commitsAfterArchival.size());
+assertFalse(commitsAfterArchival.contains(
+new HoodieInstant(State.COMPLETED,
HoodieTimeline.DELTA_COMMIT_ACTION, "0001")));
+IntStream.range(2, 10).forEach(j ->
+assertTrue(commitsAfterArchival.contains(
+new HoodieInstant(State.COMPLETED,
HoodieTimeline.DELTA_COMMIT_ACTION, "000" + j;
+ }
+}
+
+testTable.doCompaction("0010", Arrays.asList("p1", "p2"));
+
+// instant "0011" to "0019"
+for (int i = 1; i < 10; i++) {
+ testTable.doWriteOperation("001" + i, WriteOperationType.UPSERT, i
== 1
+ ? Arrays.asList("p1", "p2") : Collections.emptyList(),
Arrays.asList("p1", "p2"), 2);
+ // archival
+ Pair, List> commitsList =
archiveAndGetCommitsList(writeConfig);
+ List originalCommits = commitsList.getKey();
+ List commitsAfterArchival = commitsList.getValue();
+
+ // first 9 delta commits before the completed compaction should be
archived
+ IntStream.range(1, 10).forEach(j ->
Review comment:
The intention is to check the instants in the timeline for every commit
so that they are intact.
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/io/TestHoodieTimelineArchiver.java
##
@@ -946,6 +961,152 @@ public void
testArchiveCommitsWithCompactionCommitInMetadataTableTimeline() thro
}
}
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testArchivalWithMaxDeltaCommitsGuaranteeForCompaction(boolean
enableMetadata) throws Exception {
+HoodieWriteConfig writeConfig = initTestTableAndGetWriteConfig(
+enableMetadata, 2, 4, 8, 1, HoodieTableType.MERGE_ON_READ);
+
+// When max archival commits is set to 4, even after 8 delta commits,
since the number of delta
+// commits is still smaller than 8, the archival should not kick in.
+// The archival should only kick in after the 9th delta commit
+// instant "0001" to "0009"
+for (int i = 1; i < 10; i++) {
+ testTable.doWriteOperation("000" + i, WriteOperationType.UPSERT, i
== 1
+ ? Arrays.asList("p1", "p2") : Collections.emptyList(),
Arrays.asList("p1", "p2"), 2);
+ // archival
+ Pair, List> commitsList =
archiveAndGetCommitsList(writeConfig);
+ List originalCommits = commitsList.getKey();
+ List commitsAfterArchival = commitsList.getValue();
+
+ if (i <= 8) {
+assertEquals(originalCommits, commitsAfterArchival);
+ } else {
+assertEquals(1, originalCommits.size() - commitsAfterArchival.size());
+assertFalse(commitsAfterArchival.contains(
+new HoodieInstant(State.COMPLETED,
HoodieTimeline.DELTA_COMMIT_ACTION, "0001")));
+IntStream.range(2, 10).forEach(j ->
+assertTrue(commitsAfterArchival.contains(
+new HoodieInstant(State.COMPLETED,
HoodieTimeline.DELTA_COMMIT_ACTION, "000" + j;
+ }
+}
+
+testTable.doCompaction("0010", Arrays.asList("p1", "p2"));
+
+// instant "0011" to "0019"
+for (int i = 1; i < 10; i++) {
+ testTable.doWriteOperation("001" + i, WriteOperationType.UPSERT, i
== 1
+ ? Arrays.asList("p1", "p2") : Collections.emptyList(),
Arrays.asList("p1", "p2"), 2);
+ // archival
+ Pair, List> commitsList =
archiveAndGetCommitsList(writeConfig);
+ List
[GitHub] [hudi] yihua commented on a change in pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival
yihua commented on a change in pull request #4974:
URL: https://github.com/apache/hudi/pull/4974#discussion_r828554297
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/CompactionUtils.java
##
@@ -195,10 +196,76 @@ public static HoodieCompactionPlan
getCompactionPlan(HoodieTableMetaClient metaC
/**
* Return all pending compaction instant times.
- *
+ *
* @return
*/
public static List
getPendingCompactionInstantTimes(HoodieTableMetaClient metaClient) {
return
metaClient.getActiveTimeline().filterPendingCompactionTimeline().getInstants().collect(Collectors.toList());
}
+
+ /**
+ * Returns a pair of (timeline containing the delta commits after the latest
completed
+ * compaction commit, the completed compaction commit instant), if the
latest completed
+ * compaction commit is present; a pair of (timeline containing all the
delta commits,
+ * the first delta commit instant), if there is no completed compaction
commit.
+ *
+ * @param activeTimeline Active timeline of a table.
+ * @return Pair of timeline containing delta commits and an instant.
+ */
+ public static Option>
getDeltaCommitsSinceLatestCompaction(
+ HoodieActiveTimeline activeTimeline) {
+Option lastCompaction = activeTimeline.getCommitTimeline()
+.filterCompletedInstants().lastInstant();
+HoodieTimeline deltaCommits = activeTimeline.getDeltaCommitTimeline();
+
+HoodieInstant latestInstant;
+if (lastCompaction.isPresent()) {
+ latestInstant = lastCompaction.get();
+ // timeline containing the delta commits after the latest completed
compaction commit,
+ // and the completed compaction commit instant
+ return Option.of(Pair.of(deltaCommits.findInstantsAfter(
+ latestInstant.getTimestamp(), Integer.MAX_VALUE),
lastCompaction.get()));
+} else {
+ if (deltaCommits.countInstants() > 0) {
+latestInstant = deltaCommits.firstInstant().get();
+// timeline containing all the delta commits, and the first delta
commit instant
+return Option.of(Pair.of(deltaCommits.findInstantsAfterOrEquals(
+latestInstant.getTimestamp(), Integer.MAX_VALUE), latestInstant));
+ } else {
+return Option.empty();
+ }
+}
+ }
+
+ /**
+ * Gets the oldest instant to keep for MOR compaction.
+ * If there is no completed compaction,
+ * num delta commits >= "hoodie.compact.inline.max.delta.commits"
+ * If there is a completed compaction,
+ * num delta commits after latest completed compaction >=
"hoodie.compact.inline.max.delta.commits"
+ *
+ * @param activeTimeline Active timeline of a table.
+ * @param maxDeltaCommits Maximum number of delta commits that trigger the
compaction plan,
+ *i.e., "hoodie.compact.inline.max.delta.commits".
+ * @return the oldest instant to keep for MOR compaction.
+ */
+ public static Option getOldestInstantToKeepForCompaction(
+ HoodieActiveTimeline activeTimeline, int maxDeltaCommits) {
Review comment:
This is specific to compaction. We need to retain the deltacommits in
the timeline for compaction to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4975: [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode
hudi-bot removed a comment on pull request #4975: URL: https://github.com/apache/hudi/pull/4975#issuecomment-1069768813 ## CI report: * 0c1aad3b608f4eea6d5882b27e819ee138dd3748 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6673) * 1bc5f116d28f5ef2f6ea522a71688ea49c04b693 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4975: [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode
hudi-bot commented on pull request #4975: URL: https://github.com/apache/hudi/pull/4975#issuecomment-1069770565 ## CI report: * 0c1aad3b608f4eea6d5882b27e819ee138dd3748 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6673) * 1bc5f116d28f5ef2f6ea522a71688ea49c04b693 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7009) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival
yihua commented on a change in pull request #4974:
URL: https://github.com/apache/hudi/pull/4974#discussion_r828553673
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java
##
@@ -128,27 +129,25 @@ private HoodieCompactionPlan scheduleCompaction() {
return new HoodieCompactionPlan();
}
- private Pair getLatestDeltaCommitInfo() {
-Option lastCompaction =
table.getActiveTimeline().getCommitTimeline()
-.filterCompletedInstants().lastInstant();
-HoodieTimeline deltaCommits =
table.getActiveTimeline().getDeltaCommitTimeline();
-
-String latestInstantTs;
-final int deltaCommitsSinceLastCompaction;
-if (lastCompaction.isPresent()) {
- latestInstantTs = lastCompaction.get().getTimestamp();
- deltaCommitsSinceLastCompaction =
deltaCommits.findInstantsAfter(latestInstantTs,
Integer.MAX_VALUE).countInstants();
-} else {
- latestInstantTs = deltaCommits.firstInstant().get().getTimestamp();
- deltaCommitsSinceLastCompaction =
deltaCommits.findInstantsAfterOrEquals(latestInstantTs,
Integer.MAX_VALUE).countInstants();
+ private Option> getLatestDeltaCommitInfo() {
+Option> deltaCommitsInfo =
+
CompactionUtils.getDeltaCommitsSinceLatestCompaction(table.getActiveTimeline());
+if (deltaCommitsInfo.isPresent()) {
+ return Option.of(Pair.of(
+ deltaCommitsInfo.get().getLeft().countInstants(),
+ deltaCommitsInfo.get().getRight().getTimestamp()));
}
-return Pair.of(deltaCommitsSinceLastCompaction, latestInstantTs);
+return Option.empty();
}
private boolean needCompact(CompactionTriggerStrategy
compactionTriggerStrategy) {
boolean compactable;
// get deltaCommitsSinceLastCompaction and lastCompactionTs
-Pair latestDeltaCommitInfo = getLatestDeltaCommitInfo();
+Option> latestDeltaCommitInfoOption =
getLatestDeltaCommitInfo();
+if (!latestDeltaCommitInfoOption.isPresent()) {
+ return false;
Review comment:
This refers to the case where there is no delta commit in the timeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4975: [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode
hudi-bot removed a comment on pull request #4975: URL: https://github.com/apache/hudi/pull/4975#issuecomment-1061386882 ## CI report: * 0c1aad3b608f4eea6d5882b27e819ee138dd3748 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6673) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4975: [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode
hudi-bot commented on pull request #4975: URL: https://github.com/apache/hudi/pull/4975#issuecomment-1069768813 ## CI report: * 0c1aad3b608f4eea6d5882b27e819ee138dd3748 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6673) * 1bc5f116d28f5ef2f6ea522a71688ea49c04b693 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3649) Add HoodieTableConfig defaults to HoodieWriteConfig
[ https://issues.apache.org/jira/browse/HUDI-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3649: Priority: Major (was: Blocker) > Add HoodieTableConfig defaults to HoodieWriteConfig > --- > > Key: HUDI-3649 > URL: https://issues.apache.org/jira/browse/HUDI-3649 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.11.0 > > > HoodieWriteConfig does not set defaults from HoodieTableConfig. We need to > see if some HoodieTableConfig defaults should be set in HoodieWriteConfig. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] yihua commented on pull request #4975: [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode
yihua commented on pull request #4975: URL: https://github.com/apache/hudi/pull/4975#issuecomment-1069762836 > based on our discussion, lets add table type check and check for compaction only for MOR. bcoz, incase of Deltastreamer, no explicit compaction configs are set, but still we need to detect if async table services are enabled or not. I added the logic based on table type. By default COW table should not have any async table service enabled, while MOR table have async compaction enabled. So the auto config adjustment for metadata table is different for different types of tables. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4975: [HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode
yihua commented on a change in pull request #4975:
URL: https://github.com/apache/hudi/pull/4975#discussion_r828548624
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##
@@ -2375,19 +2375,49 @@ protected void setDefaults() {
HoodieLayoutConfig.newBuilder().fromProperties(writeConfig.getProps()).build());
writeConfig.setDefaultValue(TIMELINE_LAYOUT_VERSION_NUM,
String.valueOf(TimelineLayoutVersion.CURR_VERSION));
- // Async table services can update the metadata table and a lock
provider is
- // needed to guard against any concurrent table write operations. If
user has
- // not configured any lock provider, let's use the InProcess lock
provider.
+ autoAdjustConfigsForConcurrencyMode();
+}
+
+private void autoAdjustConfigsForConcurrencyMode() {
+ boolean isMetadataTableEnabled =
writeConfig.getBoolean(HoodieMetadataConfig.ENABLE);
final TypedProperties writeConfigProperties = writeConfig.getProps();
final boolean isLockProviderPropertySet =
writeConfigProperties.containsKey(HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME)
||
writeConfigProperties.containsKey(HoodieLockConfig.LOCK_PROVIDER_CLASS_PROP);
+
if (!isLockConfigSet) {
HoodieLockConfig.Builder lockConfigBuilder =
HoodieLockConfig.newBuilder().fromProperties(writeConfig.getProps());
-if (!isLockProviderPropertySet &&
writeConfig.areAnyTableServicesAsync()) {
- lockConfigBuilder.withLockProvider(InProcessLockProvider.class);
-}
writeConfig.setDefault(lockConfigBuilder.build());
}
+
+ if (isMetadataTableEnabled) {
+// When metadata table is enabled, optimistic concurrency control must
be used for
+// single writer with async table services.
+// Async table services can update the metadata table and a lock
provider is
+// needed to guard against any concurrent table write operations. If
user has
+// not configured any lock provider, let's use the InProcess lock
provider.
+boolean areTableServicesEnabled =
writeConfig.areTableServicesEnabled();
+boolean areAsyncTableServicesEnabled =
writeConfig.areAnyTableServicesAsync();
+
+if (!isLockProviderPropertySet && areTableServicesEnabled &&
areAsyncTableServicesEnabled) {
+ // This is targeted at Single writer with async table services
Review comment:
We need it. `areTableServicesEnabled` and
`areAsyncTableServicesEnabled` are completely independent based on the configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HUDI-3649) Add HoodieTableConfig defaults to HoodieWriteConfig
[ https://issues.apache.org/jira/browse/HUDI-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3649: Description: HoodieWriteConfig does not set defaults from HoodieTableConfig. We need to see if some HoodieTableConfig defaults should be set in HoodieWriteConfig. (was: HoodieWriteConfig does not set defaults from HoodieWriteConfig. We need to see if some config defaults should be set ) > Add HoodieTableConfig defaults to HoodieWriteConfig > --- > > Key: HUDI-3649 > URL: https://issues.apache.org/jira/browse/HUDI-3649 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > HoodieWriteConfig does not set defaults from HoodieTableConfig. We need to > see if some HoodieTableConfig defaults should be set in HoodieWriteConfig. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3649) Add HoodieTableConfig defaults to HoodieWriteConfig
[ https://issues.apache.org/jira/browse/HUDI-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3649: Description: HoodieWriteConfig does not set defaults from HoodieWriteConfig. We need to see if some config defaults should be set (was: HoodieWriteConfig does not set defaults from HoodieWriteConfig. W ) > Add HoodieTableConfig defaults to HoodieWriteConfig > --- > > Key: HUDI-3649 > URL: https://issues.apache.org/jira/browse/HUDI-3649 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > HoodieWriteConfig does not set defaults from HoodieWriteConfig. We need to > see if some config defaults should be set -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3649) Add HoodieTableConfig defaults to HoodieWriteConfig
[ https://issues.apache.org/jira/browse/HUDI-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3649: Description: HoodieWriteConfig does not set defaults from HoodieWriteConfig. W (was: HoodieWriteConfig does not set ) > Add HoodieTableConfig defaults to HoodieWriteConfig > --- > > Key: HUDI-3649 > URL: https://issues.apache.org/jira/browse/HUDI-3649 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > HoodieWriteConfig does not set defaults from HoodieWriteConfig. W -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3649) Add HoodieTableConfig defaults to HoodieWriteConfig
[ https://issues.apache.org/jira/browse/HUDI-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3649: Description: HoodieWriteConfig does not set > Add HoodieTableConfig defaults to HoodieWriteConfig > --- > > Key: HUDI-3649 > URL: https://issues.apache.org/jira/browse/HUDI-3649 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > HoodieWriteConfig does not set -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3649) Add HoodieTableConfig defaults to HoodieWriteConfig
[ https://issues.apache.org/jira/browse/HUDI-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3649: Fix Version/s: 0.11.0 > Add HoodieTableConfig defaults to HoodieWriteConfig > --- > > Key: HUDI-3649 > URL: https://issues.apache.org/jira/browse/HUDI-3649 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HUDI-3649) Add HoodieTableConfig defaults to HoodieWriteConfig
Ethan Guo created HUDI-3649: --- Summary: Add HoodieTableConfig defaults to HoodieWriteConfig Key: HUDI-3649 URL: https://issues.apache.org/jira/browse/HUDI-3649 Project: Apache Hudi Issue Type: Improvement Reporter: Ethan Guo Assignee: Ethan Guo -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4286: [WIP][HUDI-2955] Upgrade Hadoop to 3.3.1, Hive to 3.1.2, HBase to 2.4.8
hudi-bot commented on pull request #4286: URL: https://github.com/apache/hudi/pull/4286#issuecomment-1069734507 ## CI report: * dbe231115001abd287e8d948d51011b708c02f7c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6984) * c2738a5a1cd6f5ea8cffa966d7dc332f2d211f29 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7008) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4286: [WIP][HUDI-2955] Upgrade Hadoop to 3.3.1, Hive to 3.1.2, HBase to 2.4.8
hudi-bot removed a comment on pull request #4286: URL: https://github.com/apache/hudi/pull/4286#issuecomment-1069732346 ## CI report: * dbe231115001abd287e8d948d51011b708c02f7c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6984) * c2738a5a1cd6f5ea8cffa966d7dc332f2d211f29 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
