Re: [PR] [HUDI-7567] Add schema evolution to the filegroup reader [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #10957: URL: https://github.com/apache/hudi/pull/10957#issuecomment-2086001550 ## CI report: * 879e07c167692250636215e06e67b6c370496c03 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[jira] [Created] (HUDI-7693) Allow Vectorized Reading for bootstrap in the new fg reader under some conditions

2024-04-30 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-7693: - Summary: Allow Vectorized Reading for bootstrap in the new fg reader under some conditions Key: HUDI-7693 URL: https://issues.apache.org/jira/browse/HUDI-7693 Proje

Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #10352: URL: https://github.com/apache/hudi/pull/10352#issuecomment-2085972850 ## CI report: * f63dbe172cf8dec2603c266396fb7d31d5cb7f60 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[jira] [Comment Edited] (HUDI-1045) Support updates during clustering

2024-04-30 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841372#comment-17841372 ] Vinoth Chandar edited comment on HUDI-1045 at 4/30/24 4:26 PM: -

Re: [I] [SUPPORT] Hudi MOR high latency on data availability [hudi]

2024-04-30 Thread via GitHub
ad1happy2go commented on issue #8: URL: https://github.com/apache/hudi/issues/8#issuecomment-2085803990 Thanks for raising this @sgcisco . I noticed you are using compact num.delta commits as 1. Any reason for the same. If we need to compact after every commit, then better we use CO

[jira] [Updated] (HUDI-1045) Support updates during clustering

2024-04-30 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1045: - Description: We need to allow a writer w writing to file groups f1, f2, f3, concurrently while a

Re: [I] [SUPPORT] java.lang.OutOfMemoryError: Requested array size exceeds VM limit on data ingestion to COW table [hudi]

2024-04-30 Thread via GitHub
ad1happy2go commented on issue #11122: URL: https://github.com/apache/hudi/issues/11122#issuecomment-2085754426 @TarunMootala Can you check the size of the timeline files. Can you post the driver logs. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #10352: URL: https://github.com/apache/hudi/pull/10352#issuecomment-2085591638 ## CI report: * 54d5734d6d206c59c29bd8661cdef2f1692cfa49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #10352: URL: https://github.com/apache/hudi/pull/10352#issuecomment-2085573900 ## CI report: * 54d5734d6d206c59c29bd8661cdef2f1692cfa49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [SUPPORT] java.lang.OutOfMemoryError: Requested array size exceeds VM limit on data ingestion to COW table [hudi]

2024-04-30 Thread via GitHub
TarunMootala commented on issue #11122: URL: https://github.com/apache/hudi/issues/11122#issuecomment-2085549745 @ad1happy2go Thanks for your inputs. I don't think it was related to loading of archival timeline. When this error occurred, the first option I've tried is cleaning of arc

[jira] [Created] (HUDI-7692) Move MDT partiiton type code in HoodieMetadataPaylaod to MetadataPartitionType

2024-04-30 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-7692: - Summary: Move MDT partiiton type code in HoodieMetadataPaylaod to MetadataPartitionType Key: HUDI-7692 URL: https://issues.apache.org/jira/browse/HUDI-7692 Project: Apache

Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]

2024-04-30 Thread via GitHub
codope commented on code in PR #10352: URL: https://github.com/apache/hudi/pull/10352#discussion_r1584910757 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -403,6 +400,13 @@ private boolean initializeFromFiles

[jira] [Created] (HUDI-7691) Move MDT partition type related logic in HoodieBackedTableMetadataWriter to MetadataPartitionType

2024-04-30 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-7691: - Summary: Move MDT partition type related logic in HoodieBackedTableMetadataWriter to MetadataPartitionType Key: HUDI-7691 URL: https://issues.apache.org/jira/browse/HUDI-7691

Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]

2024-04-30 Thread via GitHub
codope commented on code in PR #10352: URL: https://github.com/apache/hudi/pull/10352#discussion_r1584909652 ## hudi-common/src/main/java/org/apache/hudi/common/util/BaseFileUtils.java: ## @@ -67,6 +70,61 @@ public static BaseFileUtils getInstance(HoodieFileFormat fileFormat) {

Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]

2024-04-30 Thread via GitHub
codope commented on code in PR #10352: URL: https://github.com/apache/hudi/pull/10352#discussion_r1584909257 ## hudi-common/src/main/java/org/apache/hudi/common/util/BaseFileUtils.java: ## @@ -67,6 +70,61 @@ public static BaseFileUtils getInstance(HoodieFileFormat fileFormat) {

Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]

2024-04-30 Thread via GitHub
codope commented on code in PR #10352: URL: https://github.com/apache/hudi/pull/10352#discussion_r1584905949 ## hudi-common/src/main/java/org/apache/hudi/common/util/BaseFileUtils.java: ## @@ -67,6 +70,61 @@ public static BaseFileUtils getInstance(HoodieFileFormat fileFormat) {

Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]

2024-04-30 Thread via GitHub
codope commented on code in PR #10352: URL: https://github.com/apache/hudi/pull/10352#discussion_r1584904084 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -1872,4 +1883,175 @@ public HoodieRecord next() { } }; } + + pub

Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]

2024-04-30 Thread via GitHub
codope commented on code in PR #10352: URL: https://github.com/apache/hudi/pull/10352#discussion_r1584894872 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -372,6 +379,10 @@ public static Map> convertMetada dataMetaClient

[jira] [Created] (HUDI-7690) Initialize all indexes in parallel instead of computing type by type.

2024-04-30 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-7690: - Summary: Initialize all indexes in parallel instead of computing type by type. Key: HUDI-7690 URL: https://issues.apache.org/jira/browse/HUDI-7690 Project: Apache Hudi

Re: [PR] [HUDI-7144] Build storage partition stats index and use it for data skipping [hudi]

2024-04-30 Thread via GitHub
codope commented on code in PR #10352: URL: https://github.com/apache/hudi/pull/10352#discussion_r1584882878 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -1872,4 +1883,175 @@ public HoodieRecord next() { } }; } + + pub

Re: [PR] [DOCS] Add blogs to site [hudi]

2024-04-30 Thread via GitHub
xushiyan merged PR #11127: URL: https://github.com/apache/hudi/pull/11127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apach

(hudi) branch asf-site updated: [DOCS] Add blogs to site (#11127)

2024-04-30 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new bc37698712e [DOCS] Add blogs to site (#11127

(hudi) branch asf-site updated: [DOCS] Site updates (#11128)

2024-04-30 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new e1ffc8ead6f [DOCS] Site updates (#11128) e1f

Re: [PR] [DOCS] Site updates [hudi]

2024-04-30 Thread via GitHub
xushiyan merged PR #11128: URL: https://github.com/apache/hudi/pull/11128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apach

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #11124: URL: https://github.com/apache/hudi/pull/11124#issuecomment-2085252267 ## CI report: * 33909835f589e444771c8c9c6e5bdec15785e397 UNKNOWN * d18ce474faa16547a8969cd56f67dfed5b80891a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [DOCS] Site updates [hudi]

2024-04-30 Thread via GitHub
bhasudha commented on PR #11128: URL: https://github.com/apache/hudi/pull/11128#issuecomment-2085251286 Tested locally! Home page for Github stars and Events Banner: https://github.com/apache/hudi/assets/2179254/6416638f-6865-46cf-998f-9f197fdb9787";> Add new useR in powe

[PR] [DOCS] Site updates [hudi]

2024-04-30 Thread via GitHub
bhasudha opened a new pull request, #11128: URL: https://github.com/apache/hudi/pull/11128 Summary: - Add a permanent Github star reminder at the top of the page instead of a pop up - Edit top nav so resizing does not impact sizing and spacing of top nav - Add Events Banner - Add

Re: [PR] [DOCS] Add blogs to site [hudi]

2024-04-30 Thread via GitHub
bhasudha commented on PR #11127: URL: https://github.com/apache/hudi/pull/11127#issuecomment-2085145193 Tested locally. The redirection works fine. Images are usually visible only after the merging of PR! https://github.com/apache/hudi/assets/2179254/10d40f51-038b-4ee3-b4f3-d78a5ff5fb53";

[PR] [DOCS] Add blogs to site [hudi]

2024-04-30 Thread via GitHub
bhasudha opened a new pull request, #11127: URL: https://github.com/apache/hudi/pull/11127 ### Change Logs blogs update for the site ### Impact none. Site updates ### Risk level (write none, low medium or high below) none ### Documentation Update

Re: [I] [SUPPORT] java.lang.OutOfMemoryError: Requested array size exceeds VM limit on data ingestion to COW table [hudi]

2024-04-30 Thread via GitHub
ad1happy2go commented on issue #11122: URL: https://github.com/apache/hudi/issues/11122#issuecomment-2085028466 @phani482 Is it possible for you to upgrade Hudi version to 0.14.1 and check if you still see this issue. The other issue was related to loading of archival timeline in the sync w

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #11124: URL: https://github.com/apache/hudi/pull/11124#issuecomment-2085001297 ## CI report: * 33909835f589e444771c8c9c6e5bdec15785e397 UNKNOWN * 8c1fe04e8f3c5db4683e30729770e90c45ada61e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #11124: URL: https://github.com/apache/hudi/pull/11124#issuecomment-2084988719 ## CI report: * 33909835f589e444771c8c9c6e5bdec15785e397 UNKNOWN * 8c1fe04e8f3c5db4683e30729770e90c45ada61e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7688] Stop retry inflate if encounter InterruptedIOException [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #11125: URL: https://github.com/apache/hudi/pull/11125#issuecomment-2084971701 ## CI report: * 47c838bdce65b69f6c636912036fb2e10e8f25ac Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[I] [SUPPORT] upgrade from 0.10.0 to 0.14.0 [hudi]

2024-04-30 Thread via GitHub
ghrahul opened a new issue, #11126: URL: https://github.com/apache/hudi/issues/11126 **Problem** We were running `Spark 3.2.1` along with `HUDI 0.11.1`. The jar link is: https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.2-bundle_2.12/0.11.1/hudi-spark3.2-bundle_2.12-0.11.

Re: [PR] [HUDI-7688] Stop retry inflate if encounter InterruptedIOException [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #11125: URL: https://github.com/apache/hudi/pull/11125#issuecomment-2084774561 ## CI report: * 47c838bdce65b69f6c636912036fb2e10e8f25ac Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7688] Stop retry inflate if encounter InterruptedIOException [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #11125: URL: https://github.com/apache/hudi/pull/11125#issuecomment-2084760498 ## CI report: * 47c838bdce65b69f6c636912036fb2e10e8f25ac UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #11124: URL: https://github.com/apache/hudi/pull/11124#issuecomment-2084760393 ## CI report: * 33909835f589e444771c8c9c6e5bdec15785e397 UNKNOWN * 8c1fe04e8f3c5db4683e30729770e90c45ada61e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #11124: URL: https://github.com/apache/hudi/pull/11124#issuecomment-2084744017 ## CI report: * 33909835f589e444771c8c9c6e5bdec15785e397 UNKNOWN * 8c1fe04e8f3c5db4683e30729770e90c45ada61e UNKNOWN Bot commands @hudi-bot supports the followi

Re: [I] Failure to find org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde [hudi]

2024-04-30 Thread via GitHub
liujianuuei commented on issue #160: URL: https://github.com/apache/hudi/issues/160#issuecomment-2084731565 For Chinese developers, Added following repository in the root pom.xml to fix it - works at 2024-04-30: ```xml huawei-maven https://repo.huaweicloud.co

[jira] [Updated] (HUDI-7688) Avoid always repeated inflate when encounter InterruptedIOException

2024-04-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7688: - Labels: pull-request-available (was: ) > Avoid always repeated inflate when encounter Interrupted

[PR] [HUDI-7688] Stop retry inflate if encounter InterruptedIOException [hudi]

2024-04-30 Thread via GitHub
beyond1920 opened a new pull request, #11125: URL: https://github.com/apache/hudi/pull/11125 ### Change Logs Stop retry inflate if encounter InterruptedIOException to avoid StackOverFlow error. ### Impact None ### Risk level (write none, low medium or high below)

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-04-30 Thread via GitHub
hudi-bot commented on PR #11124: URL: https://github.com/apache/hudi/pull/11124#issuecomment-2084654948 ## CI report: * 33909835f589e444771c8c9c6e5bdec15785e397 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-04-30 Thread via GitHub
danny0405 opened a new pull request, #11124: URL: https://github.com/apache/hudi/pull/11124 ### Change Logs Enable the flink metadata table by default, and the MDT with async table service can work smoothly now. ### Impact none ### Risk level (write none, low medi

Re: [I] [SUPPORT] throw "java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.client.HdfsDataInputStream.getReadStatistics() [hudi]

2024-04-30 Thread via GitHub
xicm commented on issue #1: URL: https://github.com/apache/hudi/issues/1#issuecomment-2084522466 https://hudi.apache.org/docs/troubleshooting#how-can-i-resolve-the-nosuchmethoderror-from-hbase-when-using-hudi-with-metadata-table-on-hdfs -- This is an automated message from the Apa

<    1   2