Re: [PR] [HUDI-7778] Fixing global index for duplicate updates [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11256: URL: https://github.com/apache/hudi/pull/11256#issuecomment-2118658240 ## CI report: * 89005916c14107710828a1a76d68cfa58e80bf88 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7778] Fixing global index for duplicate updates [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11256: URL: https://github.com/apache/hudi/pull/11256#issuecomment-2118644661 ## CI report: * 89005916c14107710828a1a76d68cfa58e80bf88 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7775] Remove unused APIs in HoodieStorage [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11255: URL: https://github.com/apache/hudi/pull/11255#issuecomment-2118642593 ## CI report: * 3b2ee376708bc3e71e9b310ad4f862a26c4da627 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7778] Fixing global index for duplicate updates [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11256: URL: https://github.com/apache/hudi/pull/11256#issuecomment-2118642605 ## CI report: * 89005916c14107710828a1a76d68cfa58e80bf88 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Updated] (HUDI-7778) Duplicate Key exception with RLI

2024-05-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7778: - Labels: pull-request-available (was: ) > Duplicate Key exception with RLI >

[PR] [HUDI-7778] Fixing global index for duplicate updates [hudi]

2024-05-17 Thread via GitHub
nsivabalan opened a new pull request, #11256: URL: https://github.com/apache/hudi/pull/11256 ### Change Logs We occasionally this duplicate keys being ingested to RLI partition in MDT. Fixing the root cause in this patch. Root cause: After fetching record locations from RLI

Re: [PR] [HUDI-7775] Remove unused APIs in HoodieStorage [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11255: URL: https://github.com/apache/hudi/pull/11255#issuecomment-2118630713 ## CI report: * 3b2ee376708bc3e71e9b310ad4f862a26c4da627 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7775] Remove unused APIs in HoodieStorage [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11255: URL: https://github.com/apache/hudi/pull/11255#issuecomment-2118628451 ## CI report: * 3b2ee376708bc3e71e9b310ad4f862a26c4da627 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Created] (HUDI-7778) Duplicate Key exception with RLI

2024-05-17 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-7778: - Summary: Duplicate Key exception with RLI Key: HUDI-7778 URL: https://issues.apache.org/jira/browse/HUDI-7778 Project: Apache Hudi Issue Type: Bug

[jira] [Assigned] (HUDI-7778) Duplicate Key exception with RLI

2024-05-17 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-7778: - Assignee: sivabalan narayanan > Duplicate Key exception with RLI > -

Re: [PR] [HUDI-7775] Remove unused APIs in HoodieStorage [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11255: URL: https://github.com/apache/hudi/pull/11255#issuecomment-2118613223 ## CI report: * 3b2ee376708bc3e71e9b310ad4f862a26c4da627 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7761] Make the ManifestWriter Extendable [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11253: URL: https://github.com/apache/hudi/pull/11253#issuecomment-2118613207 ## CI report: * 6d49988d2438be5710fd46e7e41af5008d4054eb Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7775] Remove unused APIs in HoodieStorage [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11255: URL: https://github.com/apache/hudi/pull/11255#issuecomment-2118602623 ## CI report: * 3b2ee376708bc3e71e9b310ad4f862a26c4da627 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7775] Remove unused APIs in HoodieStorage [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11255: URL: https://github.com/apache/hudi/pull/11255#issuecomment-2118600636 ## CI report: * 3b2ee376708bc3e71e9b310ad4f862a26c4da627 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Created] (HUDI-7777) Add function of instantiating HoodieStorage instance to meta client

2024-05-17 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-: --- Summary: Add function of instantiating HoodieStorage instance to meta client Key: HUDI- URL: https://issues.apache.org/jira/browse/HUDI- Project: Apache Hudi

[jira] [Created] (HUDI-7776) Simplify HoodieStorage instance fetching

2024-05-17 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-7776: --- Summary: Simplify HoodieStorage instance fetching Key: HUDI-7776 URL: https://issues.apache.org/jira/browse/HUDI-7776 Project: Apache Hudi Issue Type: Improvement

[jira] [Updated] (HUDI-7775) Remove unused APIs in HoodieStorage

2024-05-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7775: - Labels: pull-request-available (was: ) > Remove unused APIs in HoodieStorage > --

[PR] [HUDI-7775] Remove unused APIs in HoodieStorage [hudi]

2024-05-17 Thread via GitHub
yihua opened a new pull request, #11255: URL: https://github.com/apache/hudi/pull/11255 ### Change Logs As above. ### Impact Simplifies `HoodieStorage` APIs. ### Risk level none ### Documentation Update none ### Contributor's checklist

[jira] [Updated] (HUDI-7775) Remove unused APIs in HoodieStorage

2024-05-17 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7775: Story Points: 0.5 > Remove unused APIs in HoodieStorage > --- > >

[jira] [Created] (HUDI-7775) Remove unused APIs in HoodieStorage

2024-05-17 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-7775: --- Summary: Remove unused APIs in HoodieStorage Key: HUDI-7775 URL: https://issues.apache.org/jira/browse/HUDI-7775 Project: Apache Hudi Issue Type: Improvement

[jira] [Updated] (HUDI-7775) Remove unused APIs in HoodieStorage

2024-05-17 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7775: Fix Version/s: 0.15.0 1.0.0 > Remove unused APIs in HoodieStorage > -

[jira] [Assigned] (HUDI-7775) Remove unused APIs in HoodieStorage

2024-05-17 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7775: --- Assignee: Ethan Guo > Remove unused APIs in HoodieStorage > --- > >

Re: [PR] [HUDI-7761] Make the ManifestWriter Extendable [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11253: URL: https://github.com/apache/hudi/pull/11253#issuecomment-2118545243 ## CI report: * b035079e68c0392ec6061b31dcbba85f238bc66a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7761] Make the ManifestWriter Extendable [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11253: URL: https://github.com/apache/hudi/pull/11253#issuecomment-2118542723 ## CI report: * b035079e68c0392ec6061b31dcbba85f238bc66a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[jira] [Updated] (HUDI-6207) Files pruning for bucket index table pk filtering queries using Spark SQL

2024-05-17 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6207: - Sprint: Sprint 2023-04-26 > Files pruning for bucket index table pk filtering queries using Spark SQL > -

[jira] [Updated] (HUDI-6207) Files pruning for bucket index table pk filtering queries using Spark SQL

2024-05-17 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6207: - Reviewers: Danny Chen > Files pruning for bucket index table pk filtering queries using Spark SQL > -

Re: [PR] [HUDI-7761] Make the ManifestWriter Extendable [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11253: URL: https://github.com/apache/hudi/pull/11253#issuecomment-2118374697 ## CI report: * b035079e68c0392ec6061b31dcbba85f238bc66a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[jira] [Created] (HUDI-7774) MercifulJsonConvertor should support Avro logical type

2024-05-17 Thread Davis Zhang (Jira)
Davis Zhang created HUDI-7774: - Summary: MercifulJsonConvertor should support Avro logical type Key: HUDI-7774 URL: https://issues.apache.org/jira/browse/HUDI-7774 Project: Apache Hudi Issue Type

Re: [PR] [HUDI-7761] Make the ManifestWriter Extendable [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11253: URL: https://github.com/apache/hudi/pull/11253#issuecomment-2118228216 ## CI report: * b035079e68c0392ec6061b31dcbba85f238bc66a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7761] Make the ManifestWriter Extendable [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11253: URL: https://github.com/apache/hudi/pull/11253#issuecomment-2118219071 ## CI report: * b035079e68c0392ec6061b31dcbba85f238bc66a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[I] [SUPPORT] Fails to create a `_ro` table hive when writing table [hudi]

2024-05-17 Thread via GitHub
shubhamn21 opened a new issue, #11254: URL: https://github.com/apache/hudi/issues/11254 **Describe the problem you faced** Unable to write a hudi table to aws hadoop emr setup. From the error it seems that it is failing while creating a metadata table (with suffix `_ro`) with hive/

[jira] [Created] (HUDI-7773) Allow Users to extend S3/GCS HoodieIncrSource to bring in additional columns from upstream

2024-05-17 Thread Balaji Varadarajan (Jira)
Balaji Varadarajan created HUDI-7773: Summary: Allow Users to extend S3/GCS HoodieIncrSource to bring in additional columns from upstream Key: HUDI-7773 URL: https://issues.apache.org/jira/browse/HUDI-7773

[PR] [HUDI-7761] Changes to make Manifest Writer extendable [hudi]

2024-05-17 Thread via GitHub
csivaguru opened a new pull request, #11253: URL: https://github.com/apache/hudi/pull/11253 ### Change Logs - Change the visibility of private constructor to make it possible to extend and pluing custom manifest writer classes. - Make fetchLatestFilesForAllPartitions method in Mani

[jira] [Updated] (HUDI-7761) Make the manifest Writer Extendable

2024-05-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7761: - Labels: pull-request-available (was: ) > Make the manifest Writer Extendable > --

[I] [SUPPORT] What Class Name to use for hoodie.errortable.write.class [hudi]

2024-05-17 Thread via GitHub
soumilshah1995 opened a new issue, #11252: URL: https://github.com/apache/hudi/issues/11252 I'm trying out Hudi error tables, but I'm having trouble finding the documentation for the hoodie.errortable.write.class value. Could you please assist me? # sample config ``` hoodie.d

[jira] [Updated] (HUDI-7769) Fix Hudi CDC read with legacy parquet file format on Spark

2024-05-17 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7769: Summary: Fix Hudi CDC read with legacy parquet file format on Spark (was: Fix Hudi CDC read on Spark 3.3.4

[jira] [Updated] (HUDI-7769) Fix Hudi CDC read with legacy parquet file format on Spark

2024-05-17 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7769: Fix Version/s: 0.15.0 1.0.0 > Fix Hudi CDC read with legacy parquet file format on Spark

(hudi) branch branch-0.x updated: [MINOR] [BRANCH-0.x] Added condition to check default value to fix extracting password from credential store (#11247)

2024-05-17 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch branch-0.x in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/branch-0.x by this push: new e0cf1ce147a [MINOR] [BRANCH-0.x] Added cond

Re: [PR] [MINOR] [BRANCH-0.x] Added condition to check default value to fix extracting password from credential store [hudi]

2024-05-17 Thread via GitHub
yihua commented on PR #11247: URL: https://github.com/apache/hudi/pull/11247#issuecomment-2117886061 The CI failure is unrelated. Merging this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] [MINOR] [BRANCH-0.x] Added condition to check default value to fix extracting password from credential store [hudi]

2024-05-17 Thread via GitHub
yihua merged PR #11247: URL: https://github.com/apache/hudi/pull/11247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

(hudi) branch master updated: [MINOR] Added condition to check default value to fix extracting password from credential store (#11246)

2024-05-17 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new e4b56b090fd [MINOR] Added condition to check defaul

Re: [PR] [MINOR] Added condition to check default value to fix extracting password from credential store [hudi]

2024-05-17 Thread via GitHub
yihua merged PR #11246: URL: https://github.com/apache/hudi/pull/11246 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [MINOR] Added condition to check default value to fix extracting password from credential store [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11246: URL: https://github.com/apache/hudi/pull/11246#issuecomment-2117792852 ## CI report: * f965f6a09d5e3d70693061314b035bd93dec687b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6207] spark support bucket index query for table with bucket index [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #10191: URL: https://github.com/apache/hudi/pull/10191#issuecomment-2117790278 ## CI report: * e3223a6ef0dd865dcbd672cca9f5fb979f80ddc5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] Intermittent stall of S3 PUT request for about 17 minutes [hudi]

2024-05-17 Thread via GitHub
hgudladona commented on issue #11203: URL: https://github.com/apache/hudi/issues/11203#issuecomment-2117663830 We are mostly certain this is not due to S3 throttling but a bad socket state and its handling in the JDK 11. If you see the debug log you will notice that the socket write fails a

Re: [I] [BUG] Spark3.3 overwrite partitioned mor table failed with hudi 0.14.1 [hudi]

2024-05-17 Thread via GitHub
ad1happy2go commented on issue #10831: URL: https://github.com/apache/hudi/issues/10831#issuecomment-2117629792 @Xuehai-Chen Are you good with this? Please let us know in case you still faces error -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] [MINOR] Added condition to check default value to fix extracting password from credential store [hudi]

2024-05-17 Thread via GitHub
ad1happy2go commented on code in PR #11246: URL: https://github.com/apache/hudi/pull/11246#discussion_r1604994780 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala: ## @@ -887,7 +887,7 @@ class HoodieSparkSqlWriterInternal {

Re: [PR] [MINOR] [BRANCH-0.x] Added condition to check default value to fix extracting password from credential store [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11247: URL: https://github.com/apache/hudi/pull/11247#issuecomment-2117580619 ## CI report: * c25bdceefc761b15f50eec65b47e941e3b676916 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [MINOR] Added condition to check default value to fix extracting password from credential store [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11246: URL: https://github.com/apache/hudi/pull/11246#issuecomment-2117580492 ## CI report: * 2b979fee4a605e06c01a3a80eab2ae4aa2f4f599 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6207] spark support bucket index query for table with bucket index [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #10191: URL: https://github.com/apache/hudi/pull/10191#issuecomment-2117577092 ## CI report: * ef29826c5973ac624100b38717c685d3a1059fe2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [MINOR] [BRANCH-0.x] Added condition to check default value to fix extracting password from credential store [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11247: URL: https://github.com/apache/hudi/pull/11247#issuecomment-2117563448 ## CI report: * c25bdceefc761b15f50eec65b47e941e3b676916 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [MINOR] Added condition to check default value to fix extracting password from credential store [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11246: URL: https://github.com/apache/hudi/pull/11246#issuecomment-2117563341 ## CI report: * 2b979fee4a605e06c01a3a80eab2ae4aa2f4f599 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6207] spark support bucket index query for table with bucket index [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #10191: URL: https://github.com/apache/hudi/pull/10191#issuecomment-2117560864 ## CI report: * ef29826c5973ac624100b38717c685d3a1059fe2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6207] spark support bucket index query for table with bucket index [hudi]

2024-05-17 Thread via GitHub
KnightChess commented on code in PR #10191: URL: https://github.com/apache/hudi/pull/10191#discussion_r1604930648 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BucketIndexSupport.scala: ## @@ -0,0 +1,194 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] [HUDI-6207] spark support bucket index query for table with bucket index [hudi]

2024-05-17 Thread via GitHub
KnightChess commented on PR #10191: URL: https://github.com/apache/hudi/pull/10191#issuecomment-211750 @danny0405 yes, is ready for review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [MINOR] [BRANCH-0.x] Added condition to check default value to fix extracting password from credential store [hudi]

2024-05-17 Thread via GitHub
ad1happy2go commented on code in PR #11247: URL: https://github.com/apache/hudi/pull/11247#discussion_r1604925546 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala: ## @@ -1003,7 +1003,7 @@ class HoodieSparkSqlWriterInternal {

Re: [I] [SUPPORT] RLI index slowing down [hudi]

2024-05-17 Thread via GitHub
manishgaurav84 commented on issue #11243: URL: https://github.com/apache/hudi/issues/11243#issuecomment-2117462211 @ad1happy2go I have provided the logs on slack message. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [HUDI-5505] Fix counting of delta commits since last compaction in Sc… [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11251: URL: https://github.com/apache/hudi/pull/11251#issuecomment-2117439453 ## CI report: * 3cef36f9284541a6cad8974b2e2e9984673c6627 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [SUPPORT] Run Merge On Read Compactions [hudi]

2024-05-17 Thread via GitHub
jai20242 commented on issue #11249: URL: https://github.com/apache/hudi/issues/11249#issuecomment-2117392043 I tried it adding the configuration using compaction schedule and compaction run but it didn't work. hudi->connect --path /tmp/dep_hudi2 2024-05-17 13:25:30.737 INFO 21882

Re: [PR] [HUDI-5505] Fix counting of delta commits since last compaction in Sc… [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11251: URL: https://github.com/apache/hudi/pull/11251#issuecomment-2117319785 ## CI report: * d92e58eeaecc8b8835b317b269386fa715ca92e7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [SUPPORT] RLI index slowing down [hudi]

2024-05-17 Thread via GitHub
ad1happy2go commented on issue #11243: URL: https://github.com/apache/hudi/issues/11243#issuecomment-2117254871 @manishgaurav84 Not sure why I couldn't download event logs. Can you ping me on slack and provide me there also. -- This is an automated message from the Apache Git Service. To

Re: [I] [SUPPORT] Run Merge On Read Compactions [hudi]

2024-05-17 Thread via GitHub
ad1happy2go commented on issue #11249: URL: https://github.com/apache/hudi/issues/11249#issuecomment-2117253103 @jai20242 That is writer configuration. Hoodie don't save them. When you do compaction from cli. you need to pass there too -- This is an automated message from the

Re: [I] [SUPPORT] Hudi fails ACID verification test [hudi]

2024-05-17 Thread via GitHub
ad1happy2go commented on issue #11170: URL: https://github.com/apache/hudi/issues/11170#issuecomment-2117238692 Thanks @matthijseikelenboom for the update -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] [HUDI-5505] Fix counting of delta commits since last compaction in Sc… [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11251: URL: https://github.com/apache/hudi/pull/11251#issuecomment-2117213033 ## CI report: * d92e58eeaecc8b8835b317b269386fa715ca92e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-5505] Fix counting of delta commits since last compaction in Sc… [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11251: URL: https://github.com/apache/hudi/pull/11251#issuecomment-2117194321 ## CI report: * d92e58eeaecc8b8835b317b269386fa715ca92e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7622] Optimize HoodieTableSource's sanity check [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2117193597 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * c4a9e9a0debe32518a84877c79c4831740b95caa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [I] [SUPPORT] Run Merge On Read Compactions [hudi]

2024-05-17 Thread via GitHub
jai20242 commented on issue #11249: URL: https://github.com/apache/hudi/issues/11249#issuecomment-2117157686 I put the param hoodie.compact.inline.max.delta.commits to 1 (you can see it in the first comment) -- This is an automated message from the Apache Git Service. To respond to the me

Re: [I] [SUPPORT] Exceptions with Partition TTL [hudi]

2024-05-17 Thread via GitHub
xicm closed issue #11223: [SUPPORT] Exceptions with Partition TTL URL: https://github.com/apache/hudi/issues/11223 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

[jira] [Closed] (HUDI-7652) Add new MergeKey API to support simple and composite keys

2024-05-17 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-7652. - Resolution: Done > Add new MergeKey API to support simple and composite keys > ---

(hudi) branch master updated: [HUDI-7652] Add new `HoodieMergeKey` API to support simple and composite keys (#11077)

2024-05-17 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new e0ca6dd0d52 [HUDI-7652] Add new `HoodieMergeKey` A

Re: [PR] [HUDI-7652] Add new `HoodieMergeKey` API to support simple and composite keys [hudi]

2024-05-17 Thread via GitHub
codope merged PR #11077: URL: https://github.com/apache/hudi/pull/11077 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

Re: [PR] [HUDI-7622] Optimize HoodieTableSource's sanity check [hudi]

2024-05-17 Thread via GitHub
zhuanshenbsj1 commented on code in PR #11031: URL: https://github.com/apache/hudi/pull/11031#discussion_r1604626804 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/SanityChecks.java: ## @@ -41,23 +42,22 @@ /** * Utilities for HoodieTableFactory sanity c

Re: [PR] [HUDI-5505] Fix counting of delta commits since last compaction in Sc… [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11251: URL: https://github.com/apache/hudi/pull/11251#issuecomment-2117105740 ## CI report: * d92e58eeaecc8b8835b317b269386fa715ca92e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6207] spark support bucket index query for table with bucket index [hudi]

2024-05-17 Thread via GitHub
danny0405 commented on PR #10191: URL: https://github.com/apache/hudi/pull/10191#issuecomment-2117101176 @KnightChess Is this patch ready for review again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [HUDI-5505] Fix counting of delta commits since last compaction in Sc… [hudi]

2024-05-17 Thread via GitHub
danny0405 commented on PR #11251: URL: https://github.com/apache/hudi/pull/11251#issuecomment-2117099130 Looks reasonable, cc @nsivabalan for another look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [HUDI-7622] Optimize HoodieTableSource's sanity check [hudi]

2024-05-17 Thread via GitHub
danny0405 commented on code in PR #11031: URL: https://github.com/apache/hudi/pull/11031#discussion_r1604603189 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/SanityChecks.java: ## @@ -41,23 +42,22 @@ /** * Utilities for HoodieTableFactory sanity check

Re: [PR] [HUDI-5505] Fix counting of delta commits since last compaction in Sc… [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11251: URL: https://github.com/apache/hudi/pull/11251#issuecomment-2117092824 ## CI report: * d92e58eeaecc8b8835b317b269386fa715ca92e7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

(hudi) branch master updated (7fc5adad7aa -> d93e4eb9d70)

2024-05-17 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 7fc5adad7aa [HUDI-7717] Disable row writer for bulk insert if combining before insert is set (#11216) add d93e4

Re: [PR] [MINOR] Remove legacy code and add try catch to listStatus of partition. [hudi]

2024-05-17 Thread via GitHub
danny0405 merged PR #11250: URL: https://github.com/apache/hudi/pull/11250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apac

Re: [PR] [HUDI-7772] HoodieTimelineArchiver##getCommitInstantsToArchive need skip limiting archiving of instants [hudi]

2024-05-17 Thread via GitHub
danny0405 commented on code in PR #11245: URL: https://github.com/apache/hudi/pull/11245#discussion_r1604598008 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java: ## @@ -217,6 +217,10 @@ private List getCommitInstantsToAr

Re: [I] [SUPPORT] Hudi fails ACID verification test [hudi]

2024-05-17 Thread via GitHub
matthijseikelenboom closed issue #11170: [SUPPORT] Hudi fails ACID verification test URL: https://github.com/apache/hudi/issues/11170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] [SUPPORT] Hudi fails ACID verification test [hudi]

2024-05-17 Thread via GitHub
matthijseikelenboom commented on issue #11170: URL: https://github.com/apache/hudi/issues/11170#issuecomment-2117016086 Tested and verified. Closing issues. More info Solution has been tested on: - Java 8 ✅ - Java 11 ✅ - Java 17 ❌ (As of this moment, Hudi doesn't suppo

[jira] [Updated] (HUDI-5505) Compaction NUM_COMMITS policy should only judge completed deltacommit

2024-05-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5505: - Labels: pull-request-available (was: ) > Compaction NUM_COMMITS policy should only judge complete

[PR] [HUDI-5505] Fix counting of delta commits since last compaction in Sc… [hudi]

2024-05-17 Thread via GitHub
a-erofeev opened a new pull request, #11251: URL: https://github.com/apache/hudi/pull/11251 …heduleCompactionActionExecutor.getLatestDeltaCommitInfo ### Change Logs Fixed incorrect calculation of the number of delta commits when determining whether to schedule compaction

Re: [PR] [HUDI-7622] Optimize HoodieTableSource's sanity check [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2116997198 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * 30f50eb580ec3dec52ca87eab5a39ce027910344 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7622] Optimize HoodieTableSource's sanity check [hudi]

2024-05-17 Thread via GitHub
hudi-bot commented on PR #11031: URL: https://github.com/apache/hudi/pull/11031#issuecomment-2116984096 ## CI report: * e159472757b2475611e99dc4afd8fe2def6967f4 UNKNOWN * 30f50eb580ec3dec52ca87eab5a39ce027910344 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [I] Intermittent stall of S3 PUT request for about 17 minutes [hudi]

2024-05-17 Thread via GitHub
ad1happy2go commented on issue #11203: URL: https://github.com/apache/hudi/issues/11203#issuecomment-2116965831 @gudladona Looks like S3 throttling is happening. Did you checked if you have lot of small file groups in your data? -- This is an automated message from the Apache Git Service.

Re: [I] [SUPPORT] After upgrading hudi version 0.9.0 -> 0.13.1, it is slower and had mermory issue. [hudi]

2024-05-17 Thread via GitHub
codope closed issue #11241: [SUPPORT] After upgrading hudi version 0.9.0 -> 0.13.1, it is slower and had mermory issue. URL: https://github.com/apache/hudi/issues/11241 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] [SUPPORT] Run Merge On Read Compactions [hudi]

2024-05-17 Thread via GitHub
ad1happy2go commented on issue #11249: URL: https://github.com/apache/hudi/issues/11249#issuecomment-2116948074 @jai20242 If you have only 2 delta commits then there will be nothing to compact as default `[hoodie.compact.inline.max.delta.commits](https://hudi.apache.org/docs/configurations/