[GitHub] [hudi] boneanxs commented on a diff in pull request #6468: [HUDI-4731] Shutdown CloudWatch reporter when query completes

boneanxs commented on code in PR #6468: URL: https://github.com/apache/hudi/pull/6468#discussion_r1250338414 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala: ## @@ -594,6 +595,10 @@ object HoodieSparkSqlWriter { (syncHive

[GitHub] [hudi] nbeeee commented on issue #8965: [SUPPORT]NoSuchMethodError: org.apache.curator.CuratorZookeeperClient.startAdvancedTrace

nb commented on issue #8965: URL: https://github.com/apache/hudi/issues/8965#issuecomment-1617456302 I added the zookeeper dependency to the hudi-spark-bundle/pom.xml file and repackaged it, which is now working properly > @nb Also it this a deltastreamer job or spark datasour

[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink

flashJd commented on PR #9048: URL: https://github.com/apache/hudi/pull/9048#issuecomment-1617403962 ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

[GitHub] [hudi] danny0405 commented on issue #9075: [SUPPORT] Could you add Spark SQL Hint to be supported for hudi flexible query?

danny0405 commented on issue #9075: URL: https://github.com/apache/hudi/issues/9075#issuecomment-1617398663 A cool feature, I think we can do this, put it to the release 1.0 roadmap. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [hudi] danny0405 commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink

danny0405 commented on PR #9048: URL: https://github.com/apache/hudi/pull/9048#issuecomment-1617392915 I would prefer we fix it after 0.14.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [hudi] danny0405 commented on pull request #9058: [HUDI-6376] Support for deletes in HUDI Indexes including metadata table record index.

danny0405 commented on PR #9058: URL: https://github.com/apache/hudi/pull/9058#issuecomment-1617391263 Thanks for the feedback, we are fixing this ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [hudi] hudi-bot commented on pull request #9087: [HUDI-6329] Write pipelines for table with consistent bucket index would detect whether clustering service occurs and automatically adjust the

hudi-bot commented on PR #9087: URL: https://github.com/apache/hudi/pull/9087#issuecomment-1617378858 ## CI report: * 47e33acb50156f19b8c35eaca775f5f2ba8c5ead UNKNOWN * cc9a8531d606735d9b31b9f9f6a56f606d45b6f1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink

flashJd commented on PR #9048: URL: https://github.com/apache/hudi/pull/9048#issuecomment-1617185826 > That's true. Actually it is even more friendly for Hive query engine too, just a little late for 0.14.0 release because I'm scared for introducing protential bug, we can make the first fil

[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink

flashJd commented on PR #9048: URL: https://github.com/apache/hudi/pull/9048#issuecomment-1617185611 > That's true. Actually it is even more friendly for Hive query engine too, just a little late for 0.14.0 release because I'm scared for introducing protential bug, we can make the first fil

[GitHub] [hudi] TengHuo commented on issue #7284: [SUPPORT] Inconsistent reader and writer schema in HoodieAvroDataBlock cause exception

TengHuo commented on issue #7284: URL: https://github.com/apache/hudi/issues/7284#issuecomment-1617178498 Also fixed in PR https://github.com/apache/hudi/pull/8026 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [hudi] hudi-bot commented on pull request #9087: [HUDI-6329] Write pipelines for table with consistent bucket index would detect whether clustering service occurs and automatically adjust the

hudi-bot commented on PR #9087: URL: https://github.com/apache/hudi/pull/9087#issuecomment-1617131229 ## CI report: * 1ff671477f3635ced1643f31de4d2c47acfb3244 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1823

[GitHub] [hudi] lipusheng commented on issue #9091: [BUG] Use NonpartitionedKeyGenerator WriteOperationType BULK_INSERT and UPSERT get different _hoodie_record_key format

lipusheng commented on issue #9091: URL: https://github.com/apache/hudi/issues/9091#issuecomment-1617126765 Thanks, I will try it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[GitHub] [hudi] hudi-bot commented on pull request #9087: [HUDI-6329] Write pipelines for table with consistent bucket index would detect whether clustering service occurs and automatically adjust the

hudi-bot commented on PR #9087: URL: https://github.com/apache/hudi/pull/9087#issuecomment-1617126589 ## CI report: * 1ff671477f3635ced1643f31de4d2c47acfb3244 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1823

[GitHub] [hudi] danny0405 commented on pull request #9006: [HUDI-6404] Implement ParquetToolsExecutionStrategy for clustering

danny0405 commented on PR #9006: URL: https://github.com/apache/hudi/pull/9006#issuecomment-1617101256 The tests has passed: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=18153&view=results -- This is an automated message from the Apache Git Service. To re

[GitHub] [hudi] danny0405 commented on pull request #9038: [HUDI-6423] Incremental cleaning should consider inflight compaction instant

danny0405 commented on PR #9038: URL: https://github.com/apache/hudi/pull/9038#issuecomment-1617100426 There are many test failures: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=18249&view=logs&j=600e7de6-e133-5e69-e615-50ee129b3c08&t=bbbd7bcc-ae73-56

[GitHub] [hudi] hudi-bot commented on pull request #9087: [HUDI-6329] Write pipelines for table with consistent bucket index would detect whether clustering service occurs and automatically adjust the

hudi-bot commented on PR #9087: URL: https://github.com/apache/hudi/pull/9087#issuecomment-1617099149 ## CI report: * 1ff671477f3635ced1643f31de4d2c47acfb3244 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1823

[GitHub] [hudi] beyond1920 commented on a diff in pull request #9087: [HUDI-6329] Write pipelines for table with consistent bucket index would detect whether clustering service occurs and automaticall

beyond1920 commented on code in PR #9087: URL: https://github.com/apache/hudi/pull/9087#discussion_r125914 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/OptionsResolver.java: ## @@ -190,7 +202,18 @@ public static boolean needsAsyncClustering

[GitHub] [hudi] beyond1920 commented on a diff in pull request #9087: [HUDI-6329] Write pipelines for table with consistent bucket index would detect whether clustering service occurs and automaticall

beyond1920 commented on code in PR #9087: URL: https://github.com/apache/hudi/pull/9087#discussion_r1249074675 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/update/strategy/FlinkConsistentBucketUpdateStrategy.java: ## @@ -0,0 +1,146 @@ +/* + *

[GitHub] [hudi] hudi-bot commented on pull request #9055: [HUDI-6443] Support delete_partition, insert_overwrite/table with record-level index

hudi-bot commented on PR #9055: URL: https://github.com/apache/hudi/pull/9055#issuecomment-1617088335 ## CI report: * 05162f81e68f2f2a6364f8d8fd1143fe80b4f530 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1825

[GitHub] [hudi] hudi-bot commented on pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

hudi-bot commented on PR #9083: URL: https://github.com/apache/hudi/pull/9083#issuecomment-1617034003 ## CI report: * 3a0bfb88049cf2c0f8afe5c925dbd76fa6f7cd89 UNKNOWN * b402e3b9b1090857748665ff0628492781758b8a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #9055: [HUDI-6443] Support delete_partition, insert_overwrite/table with record-level index

hudi-bot commented on PR #9055: URL: https://github.com/apache/hudi/pull/9055#issuecomment-1616974617 ## CI report: * 58e9af49592aa04d362bb640e3310e53dd94947b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1825

[GitHub] [hudi] hudi-bot commented on pull request #9055: [HUDI-6443] Support delete_partition, insert_overwrite/table with record-level index

hudi-bot commented on PR #9055: URL: https://github.com/apache/hudi/pull/9055#issuecomment-1616898041 ## CI report: * 58e9af49592aa04d362bb640e3310e53dd94947b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1825

[GitHub] [hudi] CTTY commented on pull request #9071: [HUDI-6453] Cascade Glue schema changes to partitions

CTTY commented on PR #9071: URL: https://github.com/apache/hudi/pull/9071#issuecomment-1616868432 Hi @parisni, Sure I've attached a script to Jira to reproduce the issue: https://issues.apache.org/jira/browse/HUDI-6453 -- This is an automated message from the Apache Git Service. T

[jira] [Commented] (HUDI-6453) Cascade schema changes to partition level

[ https://issues.apache.org/jira/browse/HUDI-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739417#comment-17739417 ] Shawn Chang commented on HUDI-6453: --- I've attached `schema_reproduce.scala` script for r

[jira] [Updated] (HUDI-6453) Cascade schema changes to partition level

[ https://issues.apache.org/jira/browse/HUDI-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shawn Chang updated HUDI-6453: -- Attachment: schema_reproduce.scala > Cascade schema changes to partition level > ---

[GitHub] [hudi] parisni commented on pull request #9071: [HUDI-6453] Cascade Glue schema changes to partitions

parisni commented on PR #9071: URL: https://github.com/apache/hudi/pull/9071#issuecomment-1616843428 > the older partition is not readable due to failures described here The aws link says the error would only apply to cav/ json format. Am I missing something ? Also Athena will

[GitHub] [hudi] hudi-bot commented on pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

hudi-bot commented on PR #9083: URL: https://github.com/apache/hudi/pull/9083#issuecomment-1616830868 ## CI report: * 3a0bfb88049cf2c0f8afe5c925dbd76fa6f7cd89 UNKNOWN * 69f68c8ee2ed4cdae41cbf62a47a28b39ddcd57f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

hudi-bot commented on PR #9083: URL: https://github.com/apache/hudi/pull/9083#issuecomment-1616826707 ## CI report: * 3a0bfb88049cf2c0f8afe5c925dbd76fa6f7cd89 UNKNOWN * 69f68c8ee2ed4cdae41cbf62a47a28b39ddcd57f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] jonvex commented on a diff in pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

jonvex commented on code in PR #9083: URL: https://github.com/apache/hudi/pull/9083#discussion_r1249785136 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable3.scala: ## @@ -0,0 +1,353 @@ +/* + * Licensed to the Apache Software Foundat

[GitHub] [hudi] jonvex commented on a diff in pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

jonvex commented on code in PR #9083: URL: https://github.com/apache/hudi/pull/9083#discussion_r1249783428 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable2.scala: ## @@ -884,10 +887,9 @@ class TestMergeIntoTable2 extends HoodieSpar

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

nsivabalan commented on code in PR #9083: URL: https://github.com/apache/hudi/pull/9083#discussion_r1249752335 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala: ## @@ -125,18 +127,68 @@ case class MergeIntoHoo

[GitHub] [hudi] hudi-bot commented on pull request #9110: [MINOR] Test table cleanup

hudi-bot commented on PR #9110: URL: https://github.com/apache/hudi/pull/9110#issuecomment-1616809947 ## CI report: * 8014062978c512d904bc4d51298907f1ecdabd8d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1825

[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.

hudi-bot commented on PR #8837: URL: https://github.com/apache/hudi/pull/8837#issuecomment-1616809728 ## CI report: * 7a5df4eae6ca374734ebe19a57bb694208a0320d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1825

[GitHub] [hudi] hudi-bot commented on pull request #9110: [MINOR] Test table cleanup

hudi-bot commented on PR #9110: URL: https://github.com/apache/hudi/pull/9110#issuecomment-1616780620 ## CI report: * 8014062978c512d904bc4d51298907f1ecdabd8d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1825

[GitHub] [hudi] hudi-bot commented on pull request #9110: [MINOR] Test table cleanup

hudi-bot commented on PR #9110: URL: https://github.com/apache/hudi/pull/9110#issuecomment-1616767892 ## CI report: * 8014062978c512d904bc4d51298907f1ecdabd8d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9055: [HUDI-6443] Support delete_partition, insert_overwrite/table with record-level index

hudi-bot commented on PR #9055: URL: https://github.com/apache/hudi/pull/9055#issuecomment-1616765968 ## CI report: * 58e9af49592aa04d362bb640e3310e53dd94947b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1825

[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.

hudi-bot commented on PR #8837: URL: https://github.com/apache/hudi/pull/8837#issuecomment-1616765848 ## CI report: * 3e22656f66687bb920ec82e6764bf083985df09c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1824

[GitHub] [hudi] codope opened a new pull request, #9110: [MINOR] Test table cleanup

codope opened a new pull request, #9110: URL: https://github.com/apache/hudi/pull/9110 ### Change Logs Remove a method and reload metaclient after table config update to replace that method's usage. ### Impact none ### Risk level (write none, low medium or high be

[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.

hudi-bot commented on PR #8837: URL: https://github.com/apache/hudi/pull/8837#issuecomment-1616746845 ## CI report: * 3e22656f66687bb920ec82e6764bf083985df09c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1824

[GitHub] [hudi] jonvex commented on a diff in pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

jonvex commented on code in PR #9083: URL: https://github.com/apache/hudi/pull/9083#discussion_r1249618007 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala: ## @@ -348,7 +387,7 @@ case class MergeIntoHoodieTab

[GitHub] [hudi] hudi-bot commented on pull request #9055: [HUDI-6443] Support delete_partition, insert_overwrite/table with record-level index

hudi-bot commented on PR #9055: URL: https://github.com/apache/hudi/pull/9055#issuecomment-1616720979 ## CI report: * 6770cb108943eb86055c219a92bc1a82ffd9824a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1824

[GitHub] [hudi] jonvex commented on a diff in pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

jonvex commented on code in PR #9083: URL: https://github.com/apache/hudi/pull/9083#discussion_r1249609522 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala: ## @@ -301,15 +335,20 @@ case class MergeIntoHoodieT

[GitHub] [hudi] jonvex commented on a diff in pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

jonvex commented on code in PR #9083: URL: https://github.com/apache/hudi/pull/9083#discussion_r1249605180 ## hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark32PlusAnalysis.scala: ## @@ -128,9 +131,123 @@ case class Hoo

[GitHub] [hudi] jonvex commented on a diff in pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

jonvex commented on code in PR #9083: URL: https://github.com/apache/hudi/pull/9083#discussion_r1249604086 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/factory/HoodieSparkKeyGeneratorFactory.java: ## @@ -76,7 +76,7 @@ public class HoodieSparkKeyGenerator

[GitHub] [hudi] jonvex commented on a diff in pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

jonvex commented on code in PR #9083: URL: https://github.com/apache/hudi/pull/9083#discussion_r1249598404 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieNonIndex.java: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] [hudi] jonvex commented on a diff in pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

jonvex commented on code in PR #9083: URL: https://github.com/apache/hudi/pull/9083#discussion_r1249593464 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala: ## @@ -125,18 +127,68 @@ case class MergeIntoHoodieT

[GitHub] [hudi] jonvex commented on a diff in pull request #9083: [HUDI-6464] Spark SQL Merge Into for pkless tables

jonvex commented on code in PR #9083: URL: https://github.com/apache/hudi/pull/9083#discussion_r1249593464 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala: ## @@ -125,18 +127,68 @@ case class MergeIntoHoodieT

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9108: [HUDI-6462] Add Hudi client init callback interface

nsivabalan commented on code in PR #9108: URL: https://github.com/apache/hudi/pull/9108#discussion_r1249587516 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/callback/TestChangeConfigInitCallback.java: ## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software F

[GitHub] [hudi] hudi-bot commented on pull request #9055: [HUDI-6443] Support delete_partition, insert_overwrite/table with record-level index

hudi-bot commented on PR #9055: URL: https://github.com/apache/hudi/pull/9055#issuecomment-1616706670 ## CI report: * 6770cb108943eb86055c219a92bc1a82ffd9824a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1824

[GitHub] [hudi] gamblewin commented on issue #9093: [SUPPORT] Is it allowed using Flink Table API sqlQuery() to read data from hudi tables?

gamblewin commented on issue #9093: URL: https://github.com/apache/hudi/issues/9093#issuecomment-1616684672 @danny0405 Thx, I have reviewed the documentation on the Hudi website regarding bulk insert, and it states that bulk insert "implements a sort-based data writing algorithm", which

[GitHub] [hudi] tooptoop4 opened a new issue, #9109: [SUPPORT] performance benchmark for old version vs new version

tooptoop4 opened a new issue, #9109: URL: https://github.com/apache/hudi/issues/9109 I'm still on hudi v0.5.3 with presto as tool for reading the data. I'd be curious to see any TPC benchmark for read performance on hudi v0.5.3 vs hudi 0.12.3, is there any such graph available? this will gi

[GitHub] [hudi] hudi-bot commented on pull request #8668: [HUDI-3639] Add Proper Incremental Records FIltering support into Hudi's custom RDD

hudi-bot commented on PR #8668: URL: https://github.com/apache/hudi/pull/8668#issuecomment-1616419125 ## CI report: * 7e2708c1868584fc1828945063c160a004628b7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1825