[jira] [Updated] (HUDI-7581) Handle multi-writer index update

2024-09-26 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7581: -- Status: In Progress (was: Open) > Handle multi-writer index update > >

Re: [PR] [HUDI-6909] Use isDelete instead of isDeleted function for Spark merger [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12007: URL: https://github.com/apache/hudi/pull/12007#issuecomment-2378495638 ## CI report: * a5d28400e1e741ffa6abc2fd8ee1a40a723b4900 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=816)

Re: [PR] [HUDI-7484] Enable url encoded partitioning if hive style partition enabled [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12014: URL: https://github.com/apache/hudi/pull/12014#issuecomment-2378497506 ## CI report: * cc18dfa2fc270632b4d3d7ce545540cc98d7793b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=815)

Re: [PR] [HUDI-6909] Use isDelete instead of isDeleted function for Spark merger [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12007: URL: https://github.com/apache/hudi/pull/12007#issuecomment-2378493897 ## CI report: * 6c7a5c1051476d508716bdbdc02e5a9088c438e3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=812)

Re: [PR] [HUDI-6909] Use isDelete instead of isDeleted function for Spark merger [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12007: URL: https://github.com/apache/hudi/pull/12007#issuecomment-2378490231 ## CI report: * 6c7a5c1051476d508716bdbdc02e5a9088c438e3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=812)

Re: [PR] [HUDI-6909] Use isDelete instead of isDeleted function for Spark merger [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12007: URL: https://github.com/apache/hudi/pull/12007#issuecomment-2378488496 ## CI report: * 6c7a5c1051476d508716bdbdc02e5a9088c438e3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=812)

(hudi) branch master updated: [HUDI-7662] Add a metadata config to enable or disable functional index (#12001)

2024-09-26 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 9ca1d6c49d1 [HUDI-7662] Add a metadata config to e

Re: [PR] [HUDI-7930] Flink Support for Array of Row and Map of Row value [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11727: URL: https://github.com/apache/hudi/pull/11727#issuecomment-2378431494 ## CI report: * 53977f3241ead8f22c26372a5f88ab19bcce63fb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=814)

Re: [PR] [HUDI-7484] Enable url encoded partitioning if hive style partition enabled [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12014: URL: https://github.com/apache/hudi/pull/12014#issuecomment-2378415373 ## CI report: * d7b4b5da56dbfa628b0165e40aa8b81fd655529e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=803)

Re: [PR] [HUDI-7484] Enable url encoded partitioning if hive style partition enabled [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12014: URL: https://github.com/apache/hudi/pull/12014#issuecomment-2378414093 ## CI report: * d7b4b5da56dbfa628b0165e40aa8b81fd655529e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=803)

Re: [PR] [HUDI-8266] Ensure CleanPlanner is serializable [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12015: URL: https://github.com/apache/hudi/pull/12015#issuecomment-2378395639 ## CI report: * e15cc29fd2938eefb11446e324ed100c60b955fb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=810)

Re: [PR] [HUDI-8266] Ensure CleanPlanner is serializable [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12015: URL: https://github.com/apache/hudi/pull/12015#issuecomment-2378394526 ## CI report: * e15cc29fd2938eefb11446e324ed100c60b955fb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [I] [SUPPORT] Huge Performance Issue With BLOOM Index On A 1.6 Billion COW Table [hudi]

2024-09-26 Thread via GitHub
ad1happy2go commented on issue #11875: URL: https://github.com/apache/hudi/issues/11875#issuecomment-2378383097 @silly-carbon Did we know how much file groups the job is touching. Is it possible to attach the .hoodie zip (without metadata dir) or share one commit file to look further into i

Re: [PR] [HUDI-7930] Flink Support for Array of Row and Map of Row value [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11727: URL: https://github.com/apache/hudi/pull/11727#issuecomment-2378375324 ## CI report: * b1242af2f8445058dbafe0747c3cc9c8ee9de6de Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=807)

Re: [PR] [HUDI-7930] Flink Support for Array of Row and Map of Row value [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11727: URL: https://github.com/apache/hudi/pull/11727#issuecomment-2378374237 ## CI report: * b1242af2f8445058dbafe0747c3cc9c8ee9de6de Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=807)

[jira] [Commented] (HUDI-8220) CustomKeyGenerator can not be created with flink

2024-09-26 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885208#comment-17885208 ] Danny Chen commented on HUDI-8220: -- Moved it to 1.1.0 release because it looks like we ne

[jira] [Updated] (HUDI-8220) CustomKeyGenerator can not be created with flink

2024-09-26 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-8220: - Fix Version/s: 1.1.0 (was: 1.0.0) > CustomKeyGenerator can not be created with flin

Re: [PR] [HUDI-6909] Use isDelete instead of isDeleted function for Spark merger [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12007: URL: https://github.com/apache/hudi/pull/12007#issuecomment-2378233707 ## CI report: * c08c32127aad68da6114fbf407616cd7acba2791 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=794)

Re: [PR] [HUDI-6909] Use isDelete instead of isDeleted function for Spark merger [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12007: URL: https://github.com/apache/hudi/pull/12007#issuecomment-2378347176 ## CI report: * 6c7a5c1051476d508716bdbdc02e5a9088c438e3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=812)

Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]

2024-09-26 Thread via GitHub
codope merged PR #11921: URL: https://github.com/apache/hudi/pull/11921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[jira] [Updated] (HUDI-8266) CleanPlanner is not serializable

2024-09-26 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8266: -- Status: In Progress (was: Open) > CleanPlanner is not serializable > >

Re: [PR] [HUDI-8266] Ensure CleanPlanner is serializable [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12015: URL: https://github.com/apache/hudi/pull/12015#issuecomment-2378268551 ## CI report: * e15cc29fd2938eefb11446e324ed100c60b955fb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-8266] Ensure CleanPlanner is serializable [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12015: URL: https://github.com/apache/hudi/pull/12015#issuecomment-2378333047 ## CI report: * e15cc29fd2938eefb11446e324ed100c60b955fb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=810)

[jira] [Closed] (HUDI-8188) Add validation for partition stats index in HoodieMetadataTableValidator

2024-09-26 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-8188. - Resolution: Fixed > Add validation for partition stats index in HoodieMetadataTableValidator > ---

Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11921: URL: https://github.com/apache/hudi/pull/11921#issuecomment-2378325603 ## CI report: * c5d2797cb26fae800efd4937ba5ea33b697f5768 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=802)

Re: [PR] [DOCS] add new committer to the Team page [hudi]

2024-09-26 Thread via GitHub
wombatu-kun merged PR #12016: URL: https://github.com/apache/hudi/pull/12016 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.ap

(hudi) branch master updated: [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator (#11921)

2024-09-26 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 2994724ef5a [HUDI-8188] Add validation for partiti

Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11921: URL: https://github.com/apache/hudi/pull/11921#issuecomment-2378324740 ## CI report: * c5d2797cb26fae800efd4937ba5ea33b697f5768 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

(hudi) branch asf-site updated: [DOCS] add new committer to the Team page (#12016)

2024-09-26 Thread wombatukun
This is an automated email from the ASF dual-hosted git repository. wombatukun pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new cf52bbf624f [DOCS] add new committer to th

[jira] [Closed] (HUDI-8265) Check if we need to support functional index on bootstrap tables

2024-09-26 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-8265. - Resolution: Fixed > Check if we need to support functional index on bootstrap tables > ---

[PR] [DOCS] add new committer to the Team page [hudi]

2024-09-26 Thread via GitHub
wombatu-kun opened a new pull request, #12016: URL: https://github.com/apache/hudi/pull/12016 ### Change Logs Added new committer (wombatukun) to the Team page. ### Impact none ### Risk level (write none, low medium or high below) none ### Documentati

[jira] [Updated] (HUDI-8263) Add validation for functional index in HoodieMetadataTableValidator

2024-09-26 Thread Kate Huber (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kate Huber updated HUDI-8263: - Sprint: Hudi 1.0 Sprint2024/10/7-10/13 > Add validation for functional index in HoodieMetadataTableValidat

[jira] [Updated] (HUDI-8265) Check if we need to support functional index on bootstrap tables

2024-09-26 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8265: -- Story Points: 5 (was: 1) > Check if we need to support functional index on bootstrap tables > -

Re: [PR] [HUDI-7930] Flink Support for Array of Row and Map of Row value [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11727: URL: https://github.com/apache/hudi/pull/11727#issuecomment-2378294716 ## CI report: * b1242af2f8445058dbafe0747c3cc9c8ee9de6de Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=807)

[jira] [Reopened] (HUDI-8265) Check if we need to support functional index on bootstrap tables

2024-09-26 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reopened HUDI-8265: --- > Check if we need to support functional index on bootstrap tables > -

Re: [PR] [HUDI-6909] Use isDelete instead of isDeleted function for Spark merger [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12007: URL: https://github.com/apache/hudi/pull/12007#issuecomment-2378291805 ## CI report: * 4c37fd3c372f3f405b3070f5b9e8740be0696311 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=809)

[jira] [Updated] (HUDI-8265) Check if we need to support functional index on bootstrap tables

2024-09-26 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8265: -- Story Points: 1 > Check if we need to support functional index on bootstrap tables > ---

[jira] [Closed] (HUDI-7662) Expose a config to enable disable functional index

2024-09-26 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler closed HUDI-7662. - Resolution: Fixed > Expose a config to enable disable functional index > -

Re: [PR] [HUDI-6909] Use isDelete instead of isDeleted function for Spark merger [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12007: URL: https://github.com/apache/hudi/pull/12007#issuecomment-2378261628 ## CI report: * c08c32127aad68da6114fbf407616cd7acba2791 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=794)

[PR] [HUDI-8266] Ensure CleanPlanner is serializable [hudi]

2024-09-26 Thread via GitHub
the-other-tim-brown opened a new pull request, #12015: URL: https://github.com/apache/hudi/pull/12015 ### Change Logs - Avoids serializing a FileSystemView as part of the CleanPlanner object by leveraging the methods for the HoodieTable when needed - Avoids serializing full commit

Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]

2024-09-26 Thread via GitHub
codope merged PR #12001: URL: https://github.com/apache/hudi/pull/12001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]

2024-09-26 Thread via GitHub
codope commented on code in PR #12001: URL: https://github.com/apache/hudi/pull/12001#discussion_r1777933874 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala: ## @@ -424,6 +424,85 @@ class TestFunctionalIndex ex

Re: [PR] [HUDI-8266] Ensure CleanPlanner is serializable [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12015: URL: https://github.com/apache/hudi/pull/12015#issuecomment-2378269902 ## CI report: * e15cc29fd2938eefb11446e324ed100c60b955fb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=810)

[jira] [Updated] (HUDI-8266) CleanPlanner is not serializable

2024-09-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-8266: - Labels: pull-request-available (was: ) > CleanPlanner is not serializable > -

Re: [PR] [HUDI-6909] Use isDelete instead of isDeleted function for Spark merger [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12007: URL: https://github.com/apache/hudi/pull/12007#issuecomment-2378263143 ## CI report: * 4c37fd3c372f3f405b3070f5b9e8740be0696311 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=809)

[jira] [Updated] (HUDI-8260) Fix col stats metadata validation so that log files are also validated

2024-09-26 Thread Kate Huber (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kate Huber updated HUDI-8260: - Sprint: Hudi 1.0 Sprint2024/10/7-10/13 > Fix col stats metadata validation so that log files are also vali

[jira] [Updated] (HUDI-8262) Add validation for secondary index in HoodieMetadataTableValidator

2024-09-26 Thread Kate Huber (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kate Huber updated HUDI-8262: - Sprint: Hudi 1.0 Sprint2024/10/7-10/13 > Add validation for secondary index in HoodieMetadataTableValidato

[jira] [Assigned] (HUDI-8266) CleanPlanner is not serializable

2024-09-26 Thread Timothy Brown (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Brown reassigned HUDI-8266: --- Assignee: Timothy Brown > CleanPlanner is not serializable >

[jira] [Created] (HUDI-8266) CleanPlanner is not serializable

2024-09-26 Thread Timothy Brown (Jira)
Timothy Brown created HUDI-8266: --- Summary: CleanPlanner is not serializable Key: HUDI-8266 URL: https://issues.apache.org/jira/browse/HUDI-8266 Project: Apache Hudi Issue Type: Bug

Re: [PR] [HUDI-6909] Use isDelete instead of isDeleted function for Spark merger [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12007: URL: https://github.com/apache/hudi/pull/12007#issuecomment-2378232627 ## CI report: * c08c32127aad68da6114fbf407616cd7acba2791 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=794)

[jira] [Updated] (HUDI-3636) Clustering fails due to marker creation failure

2024-09-26 Thread Kate Huber (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kate Huber updated HUDI-3636: - Sprint: 2022/08/22, 2022/09/05, 2022/09/19, 2022/10/04, 2022/10/18, 2022/11/01, 2022/11/29, 2022/12/12, 0.

[jira] [Updated] (HUDI-3636) Clustering fails due to marker creation failure

2024-09-26 Thread Kate Huber (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kate Huber updated HUDI-3636: - Sprint: 2022/08/22, 2022/09/05, 2022/09/19, 2022/10/04, 2022/10/18, 2022/11/01, 2022/11/29, 2022/12/12, 0.

Re: [PR] [HUDI-7930] Flink Support for Array of Row and Map of Row value [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11727: URL: https://github.com/apache/hudi/pull/11727#issuecomment-2378225151 ## CI report: * b53a85428c17a226ba18c7baf1ca2a55a9f09cf8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=783)

Re: [I] [SUPPORT]problem when inserting data to a non-partitioned table created by flink sql via spark sql cli [hudi]

2024-09-26 Thread via GitHub
danny0405 commented on issue #12013: URL: https://github.com/apache/hudi/issues/12013#issuecomment-2378221641 yeah, we do have some set up logic in `HoodieTableFactory` and `HoodieHiveCatalog`, can you dig a little bit why the non-partitioned key generator is set up regardless of the explic

Re: [PR] [HUDI-7930] Flink Support for Array of Row and Map of Row value [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11727: URL: https://github.com/apache/hudi/pull/11727#issuecomment-2378224007 ## CI report: * b53a85428c17a226ba18c7baf1ca2a55a9f09cf8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=783)

Re: [PR] [HUDI-8179] Upgrade hudi flink connector to 1.20.0 [hudi]

2024-09-26 Thread via GitHub
danny0405 commented on code in PR #11966: URL: https://github.com/apache/hudi/pull/11966#discussion_r1777896860 ## pom.xml: ## @@ -139,19 +139,21 @@ 2.4.4 3.5.1 +1.19.1 1.19.1 1.18.1 1.17.1 1.16.2 1.15.1 1.14.5 -${flink1.19.v

Re: [I] [SUPPORT]problem when inserting data to a non-partitioned table created by flink sql via spark sql cli [hudi]

2024-09-26 Thread via GitHub
bithw1 commented on issue #12013: URL: https://github.com/apache/hudi/issues/12013#issuecomment-2378218107 > The non-partitioned key generator is right, did you specify the key generator on Spark side? I don't think non-partitioned key generator is right here. When I am creating the

[jira] [Resolved] (HUDI-8161) Make spark-sql command 'desc' independent from schema evolution config

2024-09-26 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-8161. -- > Make spark-sql command 'desc' independent from schema evolution config > -

[jira] [Closed] (HUDI-8161) Make spark-sql command 'desc' independent from schema evolution config

2024-09-26 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-8161. Resolution: Fixed Fixed via master branch: f2d90f5c36eb737704500b41f725b626bc7ad101 > Make spark-sql comman

Re: [PR] [HUDI-8161] Make spark-sql command 'desc' independent from schema evolution config [hudi]

2024-09-26 Thread via GitHub
danny0405 merged PR #11871: URL: https://github.com/apache/hudi/pull/11871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apac

[jira] [Updated] (HUDI-8161) Make spark-sql command 'desc' independent from schema evolution config

2024-09-26 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-8161: - Fix Version/s: 1.0.0 > Make spark-sql command 'desc' independent from schema evolution config > --

(hudi) branch master updated: [HUDI-8161] Make spark-sql command 'desc' independent from schema evolution config (#11871)

2024-09-26 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new f2d90f5c36e [HUDI-8161] Make spark-sql command

Re: [I] [SUPPORT]Schema evolution setting affects Spark's 'describe table' output [hudi]

2024-09-26 Thread via GitHub
danny0405 closed issue #11858: [SUPPORT]Schema evolution setting affects Spark's 'describe table' output URL: https://github.com/apache/hudi/issues/11858 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] [SUPPORT]problem when inserting data to a non-partitioned table created by flink sql via spark sql cli [hudi]

2024-09-26 Thread via GitHub
danny0405 commented on issue #12013: URL: https://github.com/apache/hudi/issues/12013#issuecomment-2378209962 The non-partitioned key generator is right, did you specify the key generator on Spark side? -- This is an automated message from the Apache Git Service. To respond to the message

Re: [I] [SUPPORT] Hoodie Metadata exception while upserting using Spark Structured Streaming [hudi]

2024-09-26 Thread via GitHub
danny0405 commented on issue #11997: URL: https://github.com/apache/hudi/issues/11997#issuecomment-2378206024 @dataproblems That's pretty nice~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [HUDI-8089] Remove support for Spark2 and Scala2.11 from Hudi [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11788: URL: https://github.com/apache/hudi/pull/11788#issuecomment-2378140977 ## CI report: * f75a66b6b48f3db097e6a73a90de43e74a2a5e65 UNKNOWN * 468e6f8c68f7471ddb6c35297a7c03036558e704 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47

Re: [PR] [MINOR] Add tests for SparkConfigUtils [hudi]

2024-09-26 Thread via GitHub
yihua merged PR #12000: URL: https://github.com/apache/hudi/pull/12000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

(hudi) branch master updated (c798842003c -> 55e9b45151f)

2024-09-26 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from c798842003c [HUDI-8185] Fix SPARK record for Colstats (#11969) add 55e9b45151f [MINOR] Add tests for SparkConfigUtil

Re: [I] [SUPPORT] Hoodie Metadata exception while upserting using Spark Structured Streaming [hudi]

2024-09-26 Thread via GitHub
yihua commented on issue #11997: URL: https://github.com/apache/hudi/issues/11997#issuecomment-2378096415 @dataproblems `s3a` file system implemented by `hadoop-aws` is what we suggested to use when working with Hudi tables on S3. `s3` native file system implementation is provided by EMR.

Re: [PR] [HUDI-8089] Remove support for Spark2 and Scala2.11 from Hudi [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11788: URL: https://github.com/apache/hudi/pull/11788#issuecomment-2378046718 ## CI report: * f75a66b6b48f3db097e6a73a90de43e74a2a5e65 UNKNOWN * 468e6f8c68f7471ddb6c35297a7c03036558e704 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47

Re: [PR] [HUDI-8089] Remove support for Spark2 and Scala2.11 from Hudi [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11788: URL: https://github.com/apache/hudi/pull/11788#issuecomment-2378043631 ## CI report: * f75a66b6b48f3db097e6a73a90de43e74a2a5e65 UNKNOWN * 468e6f8c68f7471ddb6c35297a7c03036558e704 UNKNOWN Bot commands @hudi-bot supports the followi

[jira] [Updated] (HUDI-8193) Support partial update in Spark Structured Streaming

2024-09-26 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-8193: -- Description: The partial update feature (added in https://github.com/apache/hudi/pull/9876) with the par

[jira] [Assigned] (HUDI-8193) Support partial update in Spark Structured Streaming

2024-09-26 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo reassigned HUDI-8193: - Assignee: Jonathan Vexler (was: Lin Liu) > Support partial update in Spark Structured Streaming

Re: [PR] [Hudi-8221] RFC82 Concurrent schema evolution detection [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12005: URL: https://github.com/apache/hudi/pull/12005#issuecomment-2377450965 ## CI report: * fbb7d0154b59120c1fa1d91094b0cffca73c2fb9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=786)

[jira] [Updated] (HUDI-7848) Fix the Comparable type of the ordering field value stored in delete record

2024-09-26 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-7848: -- Reviewers: Danny Chen, Y Ethan Guo (was: Danny Chen, Ethan Guo (this is the old account; please use "yi

Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]

2024-09-26 Thread via GitHub
jonvex commented on code in PR #12001: URL: https://github.com/apache/hudi/pull/12001#discussion_r1777675977 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala: ## @@ -424,6 +424,85 @@ class TestFunctionalIndex ex

Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]

2024-09-26 Thread via GitHub
jonvex commented on code in PR #12001: URL: https://github.com/apache/hudi/pull/12001#discussion_r1777666886 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java: ## @@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {

Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12001: URL: https://github.com/apache/hudi/pull/12001#issuecomment-2377742041 ## CI report: * c066484cc5993428a6c6a42b2c7fc2e301332be5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=804)

Re: [I] [SUPPORT] Hoodie Metadata exception while upserting using Spark Structured Streaming [hudi]

2024-09-26 Thread via GitHub
dataproblems commented on issue #11997: URL: https://github.com/apache/hudi/issues/11997#issuecomment-2377657320 @danny0405 - I am using `--packages org.apache.hudi:hudi-spark3.3-bundle_2.12:1.0.0-beta2,org.apache.hudi:hudi-aws:1.0.0-beta2` and so far I have not observed the same failure. T

Re: [PR] [HUDI-8188] Add validation for partition stats index in HoodieMetadataTableValidator [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #11921: URL: https://github.com/apache/hudi/pull/11921#issuecomment-2377602693 ## CI report: * c5d2797cb26fae800efd4937ba5ea33b697f5768 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=802)

[jira] [Updated] (HUDI-6802) Use completion time in Spark FileIndex for listing

2024-09-26 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-6802: -- Description: We need to make sure that the file index (BaseHoodieTableFileIndex, SparkHoodieTableFileIn

Re: [PR] [HUDI-7484] Enable url encoded partitioning if hive style partition enabled [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12014: URL: https://github.com/apache/hudi/pull/12014#issuecomment-2377642847 ## CI report: * d7b4b5da56dbfa628b0165e40aa8b81fd655529e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=803)

[jira] [Updated] (HUDI-7227) Enable completion time for File Group Reader

2024-09-26 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-7227: -- Description: For all query types, we should enable completion time semantic safely: # Snapshot/RO (MOR,

[jira] [Updated] (HUDI-6802) Use completion time in Spark FileIndex for listing

2024-09-26 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-6802: -- Description: We need to make sure that the file index (BaseHoodieTableFileIndex, SparkHoodieTableFileInd

[jira] [Updated] (HUDI-8017) Merge mode / write payload requires redundant configuration to work.

2024-09-26 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-8017: -- Status: Patch Available (was: In Progress) > Merge mode / write payload requires redundant configuratio

Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12001: URL: https://github.com/apache/hudi/pull/12001#issuecomment-2377593495 ## CI report: * 94aabd627f91386dd41bfb1ce1ead7b0b866bdac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797)

[jira] [Commented] (HUDI-8017) Merge mode / write payload requires redundant configuration to work.

2024-09-26 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885123#comment-17885123 ] Y Ethan Guo commented on HUDI-8017: --- This will be fixed by HUDI-8203. > Merge mode / wr

Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]

2024-09-26 Thread via GitHub
codope commented on PR #12001: URL: https://github.com/apache/hudi/pull/12001#issuecomment-2377595905 > Still have the question about: > > > In the test lines 468-479 you disable the functional index and verify the query still works. What happens if you > > > > 1. disable index

Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]

2024-09-26 Thread via GitHub
codope commented on code in PR #12001: URL: https://github.com/apache/hudi/pull/12001#discussion_r1777524200 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java: ## @@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {

Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]

2024-09-26 Thread via GitHub
codope commented on code in PR #12001: URL: https://github.com/apache/hudi/pull/12001#discussion_r1777522927 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java: ## @@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {

Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12001: URL: https://github.com/apache/hudi/pull/12001#issuecomment-2377591146 ## CI report: * 94aabd627f91386dd41bfb1ce1ead7b0b866bdac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797)

Re: [PR] [HUDI-7484] Enable url encoded partitioning if hive style partition enabled [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12014: URL: https://github.com/apache/hudi/pull/12014#issuecomment-2377512982 ## CI report: * d7b4b5da56dbfa628b0165e40aa8b81fd655529e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Updated] (HUDI-8265) Check if we need to support functional index on bootstrap tables

2024-09-26 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8265: -- Summary: Check if we need to support functional index on bootstrap tables (was: Remove usage of Path in

[jira] [Updated] (HUDI-8265) Check if we need to support functional index on bootstrap tables

2024-09-26 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8265: -- Description: [https://github.com/apache/hudi/pull/12001#discussion_r1777319335|https://github.com/apache

Re: [PR] [Hudi-8221] RFC82 Concurrent schema evolution detection [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12005: URL: https://github.com/apache/hudi/pull/12005#issuecomment-2377583219 ## CI report: * 681172b37fea38a055e87e4dc1bb953c7fa0d3c7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=801)

[jira] [Created] (HUDI-8265) Remove usage of Path in SparkMetadataWriterUtils

2024-09-26 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-8265: - Summary: Remove usage of Path in SparkMetadataWriterUtils Key: HUDI-8265 URL: https://issues.apache.org/jira/browse/HUDI-8265 Project: Apache Hudi Issue Type: Task

Re: [PR] [HUDI-7484] Enable url encoded partitioning if hive style partition enabled [hudi]

2024-09-26 Thread via GitHub
hudi-bot commented on PR #12014: URL: https://github.com/apache/hudi/pull/12014#issuecomment-2377515499 ## CI report: * d7b4b5da56dbfa628b0165e40aa8b81fd655529e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=803)

[jira] [Updated] (HUDI-7484) Fix partitioning style when partition is inferred from partitionBy

2024-09-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7484: - Labels: pull-request-available (was: ) > Fix partitioning style when partition is inferred from p

[jira] [Updated] (HUDI-7484) Fix partitioning style when partition is inferred from partitionBy

2024-09-26 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7484: -- Status: Patch Available (was: In Progress) > Fix partitioning style when partition is inferred from par

[PR] [HUDI-7484] Enable url encoded partitioning if hive style partition enabled [hudi]

2024-09-26 Thread via GitHub
codope opened a new pull request, #12014: URL: https://github.com/apache/hudi/pull/12014 ### Change Logs If hive style partitioning is enabled, then url encoding is also enabled. We need to do so otherwise the partition structure is awkward in some cases. For example, a `partition` f

  1   2   >