Re: [PR] [HUDI-7703] Clean plan to exclude partitions with no deleting file [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11136: URL: https://github.com/apache/hudi/pull/11136#issuecomment-2089765856 ## CI report: * 05e8bc658ccd29c673954a0d1e8e37d139878cc3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7703] Clean plan to exclude partitions with no deleting file [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11136: URL: https://github.com/apache/hudi/pull/11136#issuecomment-2089776154 ## CI report: * 05e8bc658ccd29c673954a0d1e8e37d139878cc3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-02 Thread via GitHub
danny0405 commented on code in PR #11124: URL: https://github.com/apache/hudi/pull/11124#discussion_r1587225060 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java: ## @@ -798,7 +795,11 @@ protected void archive(HoodieTable ta

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-02 Thread via GitHub
danny0405 commented on code in PR #11124: URL: https://github.com/apache/hudi/pull/11124#discussion_r1587227398 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java: ## @@ -558,9 +558,6 @@ protected void runTableServicesInline(

Re: [PR] [HUDI-7703] Clean plan to exclude partitions with no deleting file [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11136: URL: https://github.com/apache/hudi/pull/11136#issuecomment-2089857712 ## CI report: * f66524d2f485fc7085e7a0b4dd3d860e994a4ebf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7703] Clean plan to exclude partitions with no deleting file [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11136: URL: https://github.com/apache/hudi/pull/11136#issuecomment-2089871382 ## CI report: * f66524d2f485fc7085e7a0b4dd3d860e994a4ebf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [Feature Inquiry] index for randomized upserts [hudi]

2024-05-02 Thread via GitHub
pravin1406 closed issue #10961: [Feature Inquiry] index for randomized upserts URL: https://github.com/apache/hudi/issues/10961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [HUDI-7703] Clean plan to exclude partitions with no deleting file [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11136: URL: https://github.com/apache/hudi/pull/11136#issuecomment-2090071353 ## CI report: * 474638bd39807d6a761dda7df616d94ac82dcccb Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-02 Thread via GitHub
danny0405 merged PR #11124: URL: https://github.com/apache/hudi/pull/11124 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apac

(hudi) branch master updated (1ec7e631c38 -> 156e7604f8d)

2024-05-02 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 1ec7e631c38 [HUDI-7702] Remove unused method in ReflectUtil (#11135) add 156e7604f8d [HUDI-4372] Enable matadata

[jira] [Closed] (HUDI-4372) Enable matadata table by default for flink

2024-05-02 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-4372. Resolution: Fixed Fixed via master branch: 156e7604f8d617eb7b86bf854efb0705c49bf57d > Enable matadata table

[PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

2024-05-02 Thread via GitHub
danny0405 opened a new pull request, #11137: URL: https://github.com/apache/hudi/pull/11137 ### Change Logs Streaming ingestion pipeline requires the support because a pending instant is almost always on the timeline. ### Impact none ### Risk level (write none, lo

[jira] [Updated] (HUDI-7701) Metadata table initailization with pending instants

2024-05-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7701: - Labels: pull-request-available (was: ) > Metadata table initailization with pending instants > --

Re: [I] [SUPPORT] upgrade from 0.10.0 to 0.14.0 [hudi]

2024-05-02 Thread via GitHub
ad1happy2go commented on issue #11126: URL: https://github.com/apache/hudi/issues/11126#issuecomment-2090167198 @ghrahul Can you try to upgrade to 0.12.3 or 0.13.1 first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11137: URL: https://github.com/apache/hudi/pull/11137#issuecomment-2090230760 ## CI report: * fe7584c435e0ba03e9176cf5e7cc331d9a0052d7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11137: URL: https://github.com/apache/hudi/pull/11137#issuecomment-2090242787 ## CI report: * fe7584c435e0ba03e9176cf5e7cc331d9a0052d7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [SUPPORT] Poor Upsert Performance on COW table due to indexing [hudi]

2024-05-02 Thread via GitHub
bibhu107 commented on issue #6687: URL: https://github.com/apache/hudi/issues/6687#issuecomment-2090312515 Hi Can try https://hudi.apache.org/blog/2023/11/01/record-level-index/#metadata-table this stores the record_keys in metadata tables. But I am not sure if this indexing can be app

Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11137: URL: https://github.com/apache/hudi/pull/11137#issuecomment-2090322926 ## CI report: * fe7584c435e0ba03e9176cf5e7cc331d9a0052d7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [SUPPORT] java.lang.OutOfMemoryError: Requested array size exceeds VM limit on data ingestion to COW table [hudi]

2024-05-02 Thread via GitHub
ad1happy2go commented on issue #11122: URL: https://github.com/apache/hudi/issues/11122#issuecomment-2090370136 @TarunMootala Can you share the timeline? Do you know how many file groups are there in the clean instant? -- This is an automated message from the Apache Git Service. To respon

[jira] [Updated] (HUDI-7350) Introduce HoodieIOFactory to abstract the reader and writer implementation

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7350: Status: Open (was: In Progress) > Introduce HoodieIOFactory to abstract the reader and writer implementatio

[jira] [Assigned] (HUDI-7350) Introduce HoodieIOFactory to abstract the reader and writer implementation

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7350: --- Assignee: Jonathan Vexler (was: Ethan Guo) > Introduce HoodieIOFactory to abstract the reader and wr

[I] [SUPPORT] Merge into command fails when trying to use only few columns in source data while using partial data payload [hudi]

2024-05-02 Thread via GitHub
pravin1406 opened a new issue, #11138: URL: https://github.com/apache/hudi/issues/11138 HUDI version -> 0.14.1 Spark version -> 3.2.0 hadoop version -> 3.1.1 hive version -> 3.1.1 Hi I wanted to use partial data update payload. I have multiple sources, which all want to wr

Re: [I] [SUPPORT] java.lang.OutOfMemoryError: Requested array size exceeds VM limit on data ingestion to COW table [hudi]

2024-05-02 Thread via GitHub
TarunMootala commented on issue #11122: URL: https://github.com/apache/hudi/issues/11122#issuecomment-2090765146 @ad1happy2go > Can you share the timeline? Can you elaborate on this ? > Do you know how many file groups are there in the clean instant? Are you referring to nu

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2090782528 ## CI report: * d1de8c5240cf8f3695303a6e118538a87dea82a8 UNKNOWN * 7e38f4e8260c1bff3189873cd74dded2c012a7e2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

[jira] [Updated] (HUDI-7007) Integrate functional index using bloom filter on reader side

2024-05-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7007: -- Reviewers: Ethan Guo, Jonathan Vexler (was: Ethan Guo) > Integrate functional index using bloom filter

[jira] [Created] (HUDI-7704) Unify test client storage classes with duplicate code

2024-05-02 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-7704: - Summary: Unify test client storage classes with duplicate code Key: HUDI-7704 URL: https://issues.apache.org/jira/browse/HUDI-7704 Project: Apache Hudi Is

[jira] [Updated] (HUDI-7007) Integrate functional index using bloom filter on reader side

2024-05-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7007: -- Reviewers: Jonathan Vexler (was: Ethan Guo, Jonathan Vexler) > Integrate functional index using bloom f

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2090883624 ## CI report: * d1de8c5240cf8f3695303a6e118538a87dea82a8 UNKNOWN * 565a8c4a2d4c5e432f4a170d8993fee7a4337191 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2090899434 ## CI report: * d1de8c5240cf8f3695303a6e118538a87dea82a8 UNKNOWN * 385b66dfdce0557eea108e97320687e924b9f147 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2090939013 ## CI report: * d1de8c5240cf8f3695303a6e118538a87dea82a8 UNKNOWN * 50b743df39f533e7824741d997f0981cb2f8b32c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

[I] [SUPPORT] is it possible to read/write hudi files with another programming language? [hudi]

2024-05-02 Thread via GitHub
schlichtanders opened a new issue, #7446: URL: https://github.com/apache/hudi/issues/7446 Hi, I am curious about the state of hudi. We are currently using it via Spark, however thinking about switching to another language. Is it possible to write Hudi files via C, C++, Rust, or

Re: [I] [SUPPORT] is it possible to read/write hudi files with another programming language? [hudi]

2024-05-02 Thread via GitHub
vinothchandar commented on issue #7446: URL: https://github.com/apache/hudi/issues/7446#issuecomment-2091044457 @xushiyan do you want to share the budding hudi-rs and python bindings here, to see if anyone wants to chip in for contribtions -- This is an automated message from the Apache G

Re: [I] [SUPPORT] is it possible to read/write hudi files with another programming language? [hudi]

2024-05-02 Thread via GitHub
vinothchandar commented on issue #7446: URL: https://github.com/apache/hudi/issues/7446#issuecomment-2091046410 https://github.com/xushiyan/hudi-rs has some basic reads with datafusion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[jira] [Assigned] (HUDI-7704) Unify test client storage classes with duplicate code

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7704: --- Assignee: Vova Kolmakov > Unify test client storage classes with duplicate code > --

[jira] [Updated] (HUDI-7678) Finalize the Merger APIs and make a plan for moving over all existing built-in, custom payloads.

2024-05-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-7678: - Description: With the move towards making partial updates a first class citizen, that does not ne

[PR] [MINOR] remove unnecessary lines from java test [hudi]

2024-05-02 Thread via GitHub
jonvex opened a new pull request, #11139: URL: https://github.com/apache/hudi/pull/11139 ### Change Logs We expect it to fail in the merge handle so we do not need to validate here ### Impact less confusing code ### Risk level (write none, low medium or high below)

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091083909 ## CI report: * d1de8c5240cf8f3695303a6e118538a87dea82a8 UNKNOWN * 50b743df39f533e7824741d997f0981cb2f8b32c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [MINOR] remove unnecessary lines from java test [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11139: URL: https://github.com/apache/hudi/pull/11139#issuecomment-2091097494 ## CI report: * 069377621b3112a0280529fb15845afa9d58f991 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091097360 ## CI report: * d1de8c5240cf8f3695303a6e118538a87dea82a8 UNKNOWN * 4e4a01a12bc7022fee11414a3b6be9e72e4a18dc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1588037638 ## .github/workflows/bot.yml: ## @@ -454,17 +486,21 @@ jobs: env: FLINK_PROFILE: ${{ matrix.flinkProfile }} SPARK_PROFILE: ${{ matrix.sparkP

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091177439 ## CI report: * 4e4a01a12bc7022fee11414a3b6be9e72e4a18dc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [MINOR] remove unnecessary lines from java test [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11139: URL: https://github.com/apache/hudi/pull/11139#issuecomment-2091177535 ## CI report: * 069377621b3112a0280529fb15845afa9d58f991 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091191458 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * e869465714018ad7085a175529dfc8f700ee867c Azure: [SUCC

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091191554 ## CI report: * 4e4a01a12bc7022fee11414a3b6be9e72e4a18dc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091205464 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * e869465714018ad7085a175529dfc8f700ee867c Azure: [SUCC

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091205515 ## CI report: * 4e4a01a12bc7022fee11414a3b6be9e72e4a18dc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1588103500 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestHoodieTableValuedFunction.scala: ## @@ -689,6 +690,6 @@ class TestHoodieTableValuedFun

[jira] [Assigned] (HUDI-7705) Column name is wrong when generating partition stats index key

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7705: --- Assignee: Sagar Sumit > Column name is wrong when generating partition stats index key >

[jira] [Updated] (HUDI-7705) Column name is wrong when generating partition stats index key

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7705: Fix Version/s: 1.0.0 > Column name is wrong when generating partition stats index key >

[jira] [Created] (HUDI-7705) Column name is wrong when generating partition stats index key

2024-05-02 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-7705: --- Summary: Column name is wrong when generating partition stats index key Key: HUDI-7705 URL: https://issues.apache.org/jira/browse/HUDI-7705 Project: Apache Hudi Issue

[jira] [Updated] (HUDI-7705) Column name is wrong when generating partition stats index key

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7705: Epic Link: HUDI-512 > Column name is wrong when generating partition stats index key > -

[jira] [Updated] (HUDI-7705) Column name is wrong when generating partition stats index key

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7705: Description: When running the test "Test hudi_metadata Table-Valued Function For PARTITION_STATS index" in T

[jira] [Updated] (HUDI-7705) Column name is wrong when generating partition stats index key

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7705: Attachment: Screenshot 2024-05-02 at 11.10.59.png > Column name is wrong when generating partition stats ind

[I] [SUPPORT] ClassCastException when reading a Hudi dataset with a timestamp partition field [hudi]

2024-05-02 Thread via GitHub
harichandanp opened a new issue, #11140: URL: https://github.com/apache/hudi/issues/11140 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-s

[jira] [Created] (HUDI-7706) Improve validation in PARTITION_STATS index test

2024-05-02 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-7706: --- Summary: Improve validation in PARTITION_STATS index test Key: HUDI-7706 URL: https://issues.apache.org/jira/browse/HUDI-7706 Project: Apache Hudi Issue Type: Improvem

[jira] [Updated] (HUDI-7705) Column name is wrong when generating partition stats index key

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7705: Description: When running the test "Test hudi_metadata Table-Valued Function For PARTITION_STATS index" in

[jira] [Updated] (HUDI-7706) Improve validation in PARTITION_STATS index test

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7706: Epic Link: HUDI-512 > Improve validation in PARTITION_STATS index test > ---

[jira] [Assigned] (HUDI-7706) Improve validation in PARTITION_STATS index test

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7706: --- Assignee: Ethan Guo > Improve validation in PARTITION_STATS index test >

[jira] [Updated] (HUDI-7706) Improve validation in PARTITION_STATS index test

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7706: Fix Version/s: 1.0.0 > Improve validation in PARTITION_STATS index test > --

[jira] [Updated] (HUDI-7706) Improve validation in PARTITION_STATS index test

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7706: Description: We should add the record key in MDT when validating the partition stats. > Improve validation

[jira] [Updated] (HUDI-7705) Column name is wrong when generating partition stats index key

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7705: Description: When running the test "Test hudi_metadata Table-Valued Function For PARTITION_STATS index" in

[PR] [HUDI-7706] Improve validation in PARTITION_STATS index test [hudi]

2024-05-02 Thread via GitHub
yihua opened a new pull request, #11141: URL: https://github.com/apache/hudi/pull/11141 ### Change Logs This PR adds the validation of the record keys of the PARTITION_STATS index records in MDT when verifying the data in the PARTITION_STATS index, to make the validation robust.

[jira] [Updated] (HUDI-7706) Improve validation in PARTITION_STATS index test

2024-05-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7706: - Labels: pull-request-available (was: ) > Improve validation in PARTITION_STATS index test > -

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
jonvex commented on code in PR #11131: URL: https://github.com/apache/hudi/pull/11131#discussion_r1588149891 ## hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestUtils.java: ## @@ -68,7 +68,8 @@ public class HoodieTestUtils { public static final String[] DE

[jira] [Updated] (HUDI-7587) Move hadoop-dependent reader and writer implementation to hudi-hadoop-common module

2024-05-02 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-7587: -- Status: Patch Available (was: In Progress) > Move hadoop-dependent reader and writer implementa

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091284841 ## CI report: * b88fa88e1a946edf8da8f0686345fe06fd0f55ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091284743 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * e869465714018ad7085a175529dfc8f700ee867c Azure: [SUCC

Re: [PR] [MINOR] remove unnecessary lines from java test [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11139: URL: https://github.com/apache/hudi/pull/11139#issuecomment-2091284961 ## CI report: * 069377621b3112a0280529fb15845afa9d58f991 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
jonvex commented on code in PR #11131: URL: https://github.com/apache/hudi/pull/11131#discussion_r1588149891 ## hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestUtils.java: ## @@ -68,7 +68,8 @@ public class HoodieTestUtils { public static final String[] DE

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091302055 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * e869465714018ad7085a175529dfc8f700ee867c Azure: [SUCC

Re: [PR] [HUDI-7706] Improve validation in PARTITION_STATS index test [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11141: URL: https://github.com/apache/hudi/pull/11141#issuecomment-2091302414 ## CI report: * 3b9f0a272b58a7eb8f63ad20edd047b4aa740ccf UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091342715 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * e869465714018ad7085a175529dfc8f700ee867c Azure: [SUCC

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091343002 ## CI report: * b88fa88e1a946edf8da8f0686345fe06fd0f55ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7706] Improve validation in PARTITION_STATS index test [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11141: URL: https://github.com/apache/hudi/pull/11141#issuecomment-2091343302 ## CI report: * 3b9f0a272b58a7eb8f63ad20edd047b4aa740ccf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091448670 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * d33ea2a54ba42ccb221156b9013889b7b6b0af94 Azure: [FAIL

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091448728 ## CI report: * b88fa88e1a946edf8da8f0686345fe06fd0f55ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091461817 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * d33ea2a54ba42ccb221156b9013889b7b6b0af94 Azure: [FAIL

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091473097 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * bf6aaf244d52cc66e7c93d7a8a02502e9941 Azure: [FAIL

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091556140 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * bf6aaf244d52cc66e7c93d7a8a02502e9941 Azure: [FAIL

Re: [PR] [HUDI-7706] Improve validation in PARTITION_STATS index test [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11141: URL: https://github.com/apache/hudi/pull/11141#issuecomment-2091556714 ## CI report: * 3b9f0a272b58a7eb8f63ad20edd047b4aa740ccf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091556390 ## CI report: * c0a81f2890f9b066738fdf74cad9edf79cae0fda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

(hudi) branch master updated (156e7604f8d -> 65f4b594c28)

2024-05-02 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 156e7604f8d [HUDI-4372] Enable matadata table by default for flink (#11124) add 65f4b594c28 [HUDI-7706] Improve val

Re: [PR] [HUDI-7706] Improve validation in PARTITION_STATS index test [hudi]

2024-05-02 Thread via GitHub
yihua merged PR #11141: URL: https://github.com/apache/hudi/pull/11141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091590841 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * bf6aaf244d52cc66e7c93d7a8a02502e9941 Azure: [FAIL

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091632448 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * bf6aaf244d52cc66e7c93d7a8a02502e9941 Azure: [FAIL

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091675567 ## CI report: * c0a81f2890f9b066738fdf74cad9edf79cae0fda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [SUPPORT] is it possible to read/write hudi files with another programming language? [hudi]

2024-05-02 Thread via GitHub
xushiyan commented on issue #7446: URL: https://github.com/apache/hudi/issues/7446#issuecomment-2091776935 @vinothchandar yes. gonna take care of repo logistics and dev setup to make the repo ready for new contributors. Also preparing issues to work on. -- This is an automated message fro

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091820661 ## CI report: * c0a81f2890f9b066738fdf74cad9edf79cae0fda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091829624 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN *

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091837042 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN *

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091884462 ## CI report: * 834aad2a8b073a221e68fb3c960200f684b84dfd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091896712 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN *

[jira] [Updated] (HUDI-7706) Improve validation in PARTITION_STATS index test

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7706: Story Points: 1 > Improve validation in PARTITION_STATS index test > ---

[jira] [Updated] (HUDI-7706) Improve validation in PARTITION_STATS index test

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7706: Sprint: Sprint 2023-04-26 > Improve validation in PARTITION_STATS index test > -

[jira] [Updated] (HUDI-7702) Remove unused method in ReflectUtil

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7702: Sprint: Sprint 2023-04-26 > Remove unused method in ReflectUtil > --- > >

[jira] [Updated] (HUDI-7705) Column name is wrong when generating partition stats index key

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7705: Sprint: Sprint 2023-04-26 > Column name is wrong when generating partition stats index key > ---

[jira] [Updated] (HUDI-7695) Add docs on Spark 3.5 and Scala 2.13

2024-05-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7695: Sprint: Sprint 2023-04-26 > Add docs on Spark 3.5 and Scala 2.13 > > >

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1588541644 ## hudi-spark-datasource/hudi-spark3.5.x/src/test/java/org/apache/hudi/spark3/internal/TestReflectUtil.java: ## @@ -42,7 +44,7 @@ public void testDataSourceWriterExtraComm

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091937900 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN *

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-02 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091943286 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN *

  1   2   >