[jira] [Updated] (HUDI-7703) Clean plan does not need to include partitions with no files to delete

2024-05-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7703: - Labels: pull-request-available (was: ) > Clean plan does not need to include partitions with no

[PR] [HUDI-7703] Clean plan to exclude partitions with no deleting file [hudi]

2024-05-01 Thread via GitHub
xushiyan opened a new pull request, #11136: URL: https://github.com/apache/hudi/pull/11136 ### Change Logs Exclude partitions with no deleting files for clean plan. ### Impact Remove unnecessary info in the clean plan - minor optimization to reduce cleaner memory

[jira] [Created] (HUDI-7703) Clean plan does not need to include partitions with no files to delete

2024-05-01 Thread Raymond Xu (Jira)
Raymond Xu created HUDI-7703: Summary: Clean plan does not need to include partitions with no files to delete Key: HUDI-7703 URL: https://issues.apache.org/jira/browse/HUDI-7703 Project: Apache Hudi

(hudi) branch master updated: [HUDI-7702] Remove unused method in ReflectUtil (#11135)

2024-05-01 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 1ec7e631c38 [HUDI-7702] Remove unused method in

Re: [PR] [HUDI-7702] Remove unused method in ReflectUtil [hudi]

2024-05-01 Thread via GitHub
yihua merged PR #11135: URL: https://github.com/apache/hudi/pull/11135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-7702] Remove unused method in ReflectUtil [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11135: URL: https://github.com/apache/hudi/pull/11135#issuecomment-2089598670 ## CI report: * 07e7b3fc02030e1836161055d113795e9cf6240c Azure:

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089598608 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * e869465714018ad7085a175529dfc8f700ee867c Azure:

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2089591663 ## CI report: * d1de8c5240cf8f3695303a6e118538a87dea82a8 UNKNOWN * 7e38f4e8260c1bff3189873cd74dded2c012a7e2 Azure:

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11124: URL: https://github.com/apache/hudi/pull/11124#issuecomment-2089591532 ## CI report: * 33909835f589e444771c8c9c6e5bdec15785e397 UNKNOWN * 13d4b2235ffd4671b6573996b0f7ac3052226ad0 Azure:

Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2089584545 ## CI report: * d1de8c5240cf8f3695303a6e118538a87dea82a8 UNKNOWN * 65e6b37c7e41c84a2e37350e77e631d547dc0408 Azure:

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11124: URL: https://github.com/apache/hudi/pull/11124#issuecomment-2089584482 ## CI report: * 33909835f589e444771c8c9c6e5bdec15785e397 UNKNOWN * 13d4b2235ffd4671b6573996b0f7ac3052226ad0 Azure:

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089513705 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 5af22d4d68fb12e153472e4a2d7fffb04acb83af Azure:

Re: [PR] [HUDI-7702] Remove unused method in ReflectUtil [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11135: URL: https://github.com/apache/hudi/pull/11135#issuecomment-2089494867 ## CI report: * 07e7b3fc02030e1836161055d113795e9cf6240c Azure:

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-01 Thread via GitHub
danny0405 commented on code in PR #11124: URL: https://github.com/apache/hudi/pull/11124#discussion_r1587037847 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java: ## @@ -798,7 +795,9 @@ protected void archive(HoodieTable

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089494829 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 5af22d4d68fb12e153472e4a2d7fffb04acb83af Azure:

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-01 Thread via GitHub
danny0405 commented on code in PR #11124: URL: https://github.com/apache/hudi/pull/11124#discussion_r1587037847 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java: ## @@ -798,7 +795,9 @@ protected void archive(HoodieTable

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-01 Thread via GitHub
danny0405 commented on code in PR #11124: URL: https://github.com/apache/hudi/pull/11124#discussion_r1587036637 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/io/TestHoodieTimelineArchiver.java: ## @@ -1100,7 +1106,7 @@ public void

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-01 Thread via GitHub
danny0405 commented on code in PR #11124: URL: https://github.com/apache/hudi/pull/11124#discussion_r1587036340 ## hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java: ## @@ -289,6 +289,14 @@ public HoodieTestTable moveInflightCommitToComplete(String

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-01 Thread via GitHub
danny0405 commented on code in PR #11124: URL: https://github.com/apache/hudi/pull/11124#discussion_r1587035583 ## hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestUtils.java: ## @@ -118,6 +119,10 @@ public static StreamReadMonitoringFunction

Re: [PR] [HUDI-7702] Remove unused method in ReflectUtil [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11135: URL: https://github.com/apache/hudi/pull/11135#issuecomment-2089489284 ## CI report: * 07e7b3fc02030e1836161055d113795e9cf6240c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1587033612 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestHdfsParquetImportProcedure.scala: ## @@ -112,7 +112,7 @@ class

[jira] [Updated] (HUDI-7702) Remove unused method in ReflectUtil

2024-05-01 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7702: - Labels: pull-request-available (was: ) > Remove unused method in ReflectUtil >

[PR] [HUDI-7702] Remove unused method in ReflectUtil [hudi]

2024-05-01 Thread via GitHub
yihua opened a new pull request, #11135: URL: https://github.com/apache/hudi/pull/11135 ### Change Logs `ReflectUtil#createInsertInto` is no longer used in the repo and causes an issue for Scala 2.13 support. We should remove the unused method. ### Impact Code cleanup.

[jira] [Updated] (HUDI-7702) Remove unused method in ReflectUtil

2024-05-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7702: Description: ReflectUtil#createInsertInto is no longer used in the repo and causes issue for Scala 2.13

[jira] [Updated] (HUDI-7702) Remove unused method in ReflectUtil

2024-05-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7702: Description: createInsertInto > Remove unused method in ReflectUtil > --- >

[jira] [Created] (HUDI-7702) Remove unused method in ReflectUtil

2024-05-01 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-7702: --- Summary: Remove unused method in ReflectUtil Key: HUDI-7702 URL: https://issues.apache.org/jira/browse/HUDI-7702 Project: Apache Hudi Issue Type: Improvement

[jira] [Updated] (HUDI-7702) Remove unused method in ReflectUtil

2024-05-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7702: Fix Version/s: 0.15.0 1.0.0 > Remove unused method in ReflectUtil >

[jira] [Closed] (HUDI-7694) Unify bijection-avro dependency version

2024-05-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-7694. --- Resolution: Fixed > Unify bijection-avro dependency version > --- > >

[jira] [Assigned] (HUDI-7702) Remove unused method in ReflectUtil

2024-05-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7702: --- Assignee: Ethan Guo > Remove unused method in ReflectUtil > --- > >

Re: [PR] [HUDI-4372] Enable matadata table by default for flink [hudi]

2024-05-01 Thread via GitHub
codope commented on code in PR #11124: URL: https://github.com/apache/hudi/pull/11124#discussion_r1587021674 ## hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestUtils.java: ## @@ -118,6 +119,10 @@ public static StreamReadMonitoringFunction

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1587020774 ## .github/workflows/bot.yml: ## @@ -454,17 +486,21 @@ jobs: env: FLINK_PROFILE: ${{ matrix.flinkProfile }} SPARK_PROFILE: ${{

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089452939 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 43488ee2970b0680b63a212b7c2652bd717cb0db Azure:

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089446845 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9196766e914173f0aa16aa57ca79da036a296dbb Azure:

[jira] [Updated] (HUDI-7665) Rolling upgrade of 1.0

2024-05-01 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-7665: - Description: We need to update the table version due to the format changes in 1.0. | * Plan to

[jira] [Assigned] (HUDI-7665) Rolling upgrade of 1.0

2024-05-01 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-7665: Assignee: Balaji Varadarajan > Rolling upgrade of 1.0 > --- > >

[jira] [Updated] (HUDI-7665) Rolling upgrade of 1.0

2024-05-01 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-7665: - Summary: Rolling upgrade of 1.0 (was: Upgrade Table Version) > Rolling upgrade of 1.0 >

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1587003129 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/util/JavaScalaConverters.scala: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1587002563 ## packaging/bundle-validation/base/Dockerfile: ## @@ -51,9 +52,16 @@ RUN wget https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK && rm

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1587001638 ## hudi-spark-datasource/hudi-spark3.5.x/src/test/java/org/apache/hudi/spark3/internal/TestReflectUtil.java: ## @@ -42,7 +44,7 @@ public void

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1587001209 ## hudi-spark-datasource/hudi-spark3-common/src/main/java/org/apache/hudi/spark3/internal/ReflectUtil.java: ## @@ -23,7 +23,7 @@ import

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586999412 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestHdfsParquetImportProcedure.scala: ## @@ -112,7 +112,7 @@ class

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586998696 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/analysis/TestHoodiePruneFileSourcePartitions.scala: ## @@ -107,12 +107,12 @@ class

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586999074 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestHoodieTableValuedFunction.scala: ## @@ -689,6 +690,6 @@ class

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586998232 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala: ## @@ -959,7 +959,7 @@ class TestCOWDataSource extends

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
jonvex commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586997450 ## .github/workflows/bot.yml: ## @@ -454,17 +486,21 @@ jobs: env: FLINK_PROFILE: ${{ matrix.flinkProfile }} SPARK_PROFILE: ${{

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586996825 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala: ## @@ -959,7 +959,7 @@ class TestCOWDataSource extends

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586996341 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ValidateMetadataTableFilesProcedure.scala: ## @@ -115,10 +115,10 @@ class

Re: [I] [SUPPORT] java.lang.ClassCastException: class org.apache.spark.sql.catalyst.expressions.UnsafeRow cannot be cast to class org.apache.spark.sql.vectorized.ColumnarBatch [hudi]

2024-05-01 Thread via GitHub
vicuna96 commented on issue #11106: URL: https://github.com/apache/hudi/issues/11106#issuecomment-2089425319 Hi @danny0405 , this seems to be in the spark-catalyst_2.12-3.3.2.jar package. but org.apache.spark.sql.catalyst.expressions.UnsafeRow does not extend

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586994861 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ExportInstantsProcedure.scala: ## @@ -176,12 +176,12 @@ class

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586993646 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala: ## @@ -263,10 +262,10 @@ object DefaultSource { Option(schema)

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586992547 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/ColumnStatsIndexSupport.scala: ## @@ -308,7 +309,7 @@ class ColumnStatsIndexSupport(spark:

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586990669 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieInternalRowUtils.scala: ## @@ -18,11 +18,12 @@ package org.apache.spark.sql -import

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586989393 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala: ## @@ -241,17 +241,16 @@ object HoodieDatasetBulkInsertHelper }

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r158697 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieConversionUtils.scala: ## @@ -30,9 +31,7 @@ object HoodieConversionUtils { * a mutable one)

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586988569 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala: ## @@ -18,20 +18,20 @@ package org.apache.hudi -import

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089411018 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9196766e914173f0aa16aa57ca79da036a296dbb Azure:

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
yihua commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586987506 ## .github/workflows/bot.yml: ## @@ -454,17 +486,21 @@ jobs: env: FLINK_PROFILE: ${{ matrix.flinkProfile }} SPARK_PROFILE: ${{

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089405801 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9196766e914173f0aa16aa57ca79da036a296dbb Azure:

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089400409 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9196766e914173f0aa16aa57ca79da036a296dbb Azure:

[jira] [Commented] (HUDI-6495) Finalize the RFC-61/Non-blocking Concurrency Control design

2024-05-01 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842818#comment-17842818 ] Danny Chen commented on HUDI-6495: -- The MDT compaction is already switched to NBCC style now, which can

[jira] [Updated] (HUDI-4372) Enable matadata table by default for flink

2024-05-01 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-4372: - Status: Patch Available (was: In Progress) > Enable matadata table by default for flink >

[jira] [Updated] (HUDI-4372) Enable matadata table by default for flink

2024-05-01 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-4372: - Reviewers: Ethan Guo > Enable matadata table by default for flink >

[jira] [Updated] (HUDI-6296) Add Scala 2.13 build profile to support scala 2.13

2024-05-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6296: Reviewers: Jonathan Vexler > Add Scala 2.13 build profile to support scala 2.13 >

[jira] [Updated] (HUDI-6296) Add Scala 2.13 build profile to support scala 2.13

2024-05-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6296: Status: Patch Available (was: In Progress) > Add Scala 2.13 build profile to support scala 2.13 >

[jira] [Updated] (HUDI-7701) Metadata table initailization with pending instants

2024-05-01 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-7701: - Sprint: Sprint 2023-04-26 > Metadata table initailization with pending instants >

[jira] [Closed] (HUDI-7672) Fix the Hive server scratch dir for tests in hudi-utilities

2024-05-01 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-7672. Resolution: Fixed > Fix the Hive server scratch dir for tests in hudi-utilities >

[jira] [Created] (HUDI-7701) Metadata table initailization with pending instants

2024-05-01 Thread Danny Chen (Jira)
Danny Chen created HUDI-7701: Summary: Metadata table initailization with pending instants Key: HUDI-7701 URL: https://issues.apache.org/jira/browse/HUDI-7701 Project: Apache Hudi Issue Type:

[jira] [Updated] (HUDI-7701) Metadata table initailization with pending instants

2024-05-01 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-7701: - Status: In Progress (was: Open) > Metadata table initailization with pending instants >

[jira] [Closed] (HUDI-7633) Use try with resources for AutoCloseable

2024-05-01 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-7633. --- Resolution: Fixed > Use try with resources for AutoCloseable > > >

[jira] [Updated] (HUDI-7538) Consolidate the CDC Formats (changelog format, RFC-51)

2024-05-01 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-7538: - Description: For sake of more consistency, we need to consolidate the the changelog mode

[jira] [Updated] (HUDI-7538) Consolidate the CDC Formats (changelog format, RFC-51)

2024-05-01 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-7538: - Reviewers: Danny Chen, Ethan Guo > Consolidate the CDC Formats (changelog format, RFC-51) >

[jira] [Updated] (HUDI-7538) Consolidate the CDC Formats (changelog format, RFC-51)

2024-05-01 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-7538: - Description: For sake of more consistency, we need to consolidate the the changelog mode

[jira] [Updated] (HUDI-7538) Consolidate the CDC Formats (changelog format, RFC-51)

2024-05-01 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-7538: - Description: For sake of more consistency, we need to consolidate the the changelog mode

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
jonvex commented on code in PR #11130: URL: https://github.com/apache/hudi/pull/11130#discussion_r1586941487 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieConversionUtils.scala: ## @@ -30,9 +31,7 @@ object HoodieConversionUtils { * a mutable one)

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089355649 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * ef6d315d941cc770a3212fe7530294fdec30f749 Azure:

[jira] [Updated] (HUDI-7700) Support query hint to inject indexes in query plans

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7700: -- Sprint: Sprint 2023-04-26 > Support query hint to inject indexes in query plans >

[jira] [Updated] (HUDI-7700) Support query hint to inject indexes in query plans

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7700: -- Story Points: 6 > Support query hint to inject indexes in query plans >

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089320198 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * ef6d315d941cc770a3212fe7530294fdec30f749 Azure:

[jira] [Created] (HUDI-7700) Support query hint to inject indexes in query plans

2024-05-01 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-7700: - Summary: Support query hint to inject indexes in query plans Key: HUDI-7700 URL: https://issues.apache.org/jira/browse/HUDI-7700 Project: Apache Hudi Issue Type:

[jira] [Updated] (HUDI-7661) Create index readme to show how a new index implementation can be added

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7661: -- Summary: Create index readme to show how a new index implementation can be added (was: Update docs to

[jira] [Updated] (HUDI-7661) Update docs to show how a new index implementation can be added

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7661: -- Story Points: 0.5 (was: 1) > Update docs to show how a new index implementation can be added >

[jira] [Updated] (HUDI-7661) Update docs to show how a new index implementation can be added

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7661: -- Sprint: Sprint 2023-04-26 > Update docs to show how a new index implementation can be added >

Re: [PR] [HUDI-3304] Add support for selective partial update [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #9979: URL: https://github.com/apache/hudi/pull/9979#issuecomment-2089312742 ## CI report: * b038e47bc8365959cc7d9a4a4d5fe07e081dd64e UNKNOWN * 5ea3b0b905186b2701ee57f466cbec82043ddbea Azure:

[jira] [Assigned] (HUDI-7661) Update docs to show how a new index implementation can be added

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-7661: - Assignee: Sagar Sumit > Update docs to show how a new index implementation can be added >

[jira] [Assigned] (HUDI-7696) Consolidate convertFilesToPartitionStatsRecords and convertMetadataToPartitionStatsRecords

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-7696: - Assignee: Sagar Sumit > Consolidate convertFilesToPartitionStatsRecords and >

[jira] [Updated] (HUDI-7661) Update docs to show how a new index implementation can be added

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7661: -- Story Points: 1 > Update docs to show how a new index implementation can be added >

[jira] [Assigned] (HUDI-7691) Move MDT partition type related logic in HoodieBackedTableMetadataWriter to MetadataPartitionType

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-7691: - Assignee: Sagar Sumit > Move MDT partition type related logic in HoodieBackedTableMetadataWriter

[jira] [Updated] (HUDI-7692) Move MDT partiiton type code in HoodieMetadataPaylaod to MetadataPartitionType

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7692: -- Sprint: Sprint 2023-04-26 > Move MDT partiiton type code in HoodieMetadataPaylaod to

[jira] [Updated] (HUDI-7696) Consolidate convertFilesToPartitionStatsRecords and convertMetadataToPartitionStatsRecords

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7696: -- Sprint: Sprint 2023-04-26 > Consolidate convertFilesToPartitionStatsRecords and >

[jira] [Updated] (HUDI-7691) Move MDT partition type related logic in HoodieBackedTableMetadataWriter to MetadataPartitionType

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7691: -- Sprint: Sprint 2023-04-26 > Move MDT partition type related logic in HoodieBackedTableMetadataWriter to

[jira] [Assigned] (HUDI-7692) Move MDT partiiton type code in HoodieMetadataPaylaod to MetadataPartitionType

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-7692: - Assignee: Sagar Sumit > Move MDT partiiton type code in HoodieMetadataPaylaod to

[jira] [Updated] (HUDI-7690) Initialize all indexes in parallel instead of computing type by type.

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7690: -- Story Points: 2 > Initialize all indexes in parallel instead of computing type by type. >

[jira] [Updated] (HUDI-7662) Expose a config to enable disable functional index

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7662: -- Story Points: 1 > Expose a config to enable disable functional index >

[jira] [Updated] (HUDI-7696) Consolidate convertFilesToPartitionStatsRecords and convertMetadataToPartitionStatsRecords

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7696: -- Story Points: 1 > Consolidate convertFilesToPartitionStatsRecords and >

[jira] [Updated] (HUDI-7691) Move MDT partition type related logic in HoodieBackedTableMetadataWriter to MetadataPartitionType

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7691: -- Story Points: 2 > Move MDT partition type related logic in HoodieBackedTableMetadataWriter to >

[jira] [Updated] (HUDI-7692) Move MDT partiiton type code in HoodieMetadataPaylaod to MetadataPartitionType

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7692: -- Story Points: 1 > Move MDT partiiton type code in HoodieMetadataPaylaod to MetadataPartitionType >

Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2089307547 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * ef6d315d941cc770a3212fe7530294fdec30f749 Azure:

[jira] [Updated] (HUDI-7007) Integrate functional index using bloom filter on reader side

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7007: -- Reviewers: Ethan Guo > Integrate functional index using bloom filter on reader side >

Re: [PR] [HUDI-3304] Add support for selective partial update [hudi]

2024-05-01 Thread via GitHub
hudi-bot commented on PR #9979: URL: https://github.com/apache/hudi/pull/9979#issuecomment-2089306423 ## CI report: * b038e47bc8365959cc7d9a4a4d5fe07e081dd64e UNKNOWN * e4d364ea34d09041a74086afc7804c19574a Azure:

[jira] [Closed] (HUDI-7144) Support query for tables written as partitionBy but synced as non-partitioned

2024-05-01 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-7144. - Resolution: Done > Support query for tables written as partitionBy but synced as non-partitioned >

  1   2   >