[GitHub] [hudi] hudi-bot commented on pull request #9731: [MINOR] Add tests on combine parallelism

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9731: URL: https://github.com/apache/hudi/pull/9731#issuecomment-1722142867 ## CI report: * 047941b66ee52a99f626fd0dadb72581d9855385 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9730: [HUDI-6870] Pass project ID to job

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9730: URL: https://github.com/apache/hudi/pull/9730#issuecomment-1722132978 ## CI report: * 498fd3fb4586fba8a2dd5e66da41d9d8cf11a3d5 UNKNOWN * 0490cca360c6542fd52ac3316b8f2047386654fd Azure:

[GitHub] [hudi] Jason-liujc commented on issue #9728: [SUPPORT] Hudi Job fails fast in concurrent write even with high retries and long wait time

2023-09-15 Thread via GitHub
Jason-liujc commented on issue #9728: URL: https://github.com/apache/hudi/issues/9728#issuecomment-1722132706 Thanks! That's good to know. Meanwhile we can build our own "lock" However, does this symptom also mean these retry parameters does not work for DynamoDB based lock provider?

[jira] [Updated] (HUDI-6861) Update SQL Pages for 0.14.0

2023-09-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6861: - Labels: pull-request-available (was: ) > Update SQL Pages for 0.14.0 >

[hudi] branch asf-site updated: [HUDI-6861] update sql pages for 0.14.0 (#9699)

2023-09-15 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new e87bf7c26c6 [HUDI-6861] update sql pages

[GitHub] [hudi] nsivabalan merged pull request #9699: [HUDI-6861] update sql pages for 0.14.0

2023-09-15 Thread via GitHub
nsivabalan merged PR #9699: URL: https://github.com/apache/hudi/pull/9699 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot commented on pull request #9730: [HUDI-6870] Pass project ID to job

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9730: URL: https://github.com/apache/hudi/pull/9730#issuecomment-1722111228 ## CI report: * e998928a33653fb70a7a16e86b141c3077659214 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9731: [MINOR] Add tests on combine parallelism

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9731: URL: https://github.com/apache/hudi/pull/9731#issuecomment-1722111236 ## CI report: * 047941b66ee52a99f626fd0dadb72581d9855385 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9730: [HUDI-6870] Pass project ID to job

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9730: URL: https://github.com/apache/hudi/pull/9730#issuecomment-1722101349 ## CI report: * e998928a33653fb70a7a16e86b141c3077659214 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9731: [MINOR] Add tests on combine parallelism

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9731: URL: https://github.com/apache/hudi/pull/9731#issuecomment-1722101360 ## CI report: * 047941b66ee52a99f626fd0dadb72581d9855385 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] bhasudha commented on a diff in pull request #9709: [HUDI-6856] Adding a page for partially failed commits

2023-09-15 Thread via GitHub
bhasudha commented on code in PR #9709: URL: https://github.com/apache/hudi/pull/9709#discussion_r1327886719 ## website/docs/rollbacks.md: ## @@ -0,0 +1,67 @@ +--- +title: Partially Failed Commits Review Comment: Propose a title like "Rollback nechansim" since this page is

[GitHub] [hudi] gtk96 opened a new issue, #9732: [SUPPORT] Executor executes action [commits the instant 20230916074105355] error

2023-09-15 Thread via GitHub
gtk96 opened a new issue, #9732: URL: https://github.com/apache/hudi/issues/9732 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? yes - Join the mailing list to engage in conversations and get faster support at

[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata

2023-09-15 Thread via GitHub
danny0405 commented on code in PR #9651: URL: https://github.com/apache/hudi/pull/9651#discussion_r1327886335 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/util/TimelineServerHelper.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [hudi] hudi-bot commented on pull request #9730: [HUDI-6870] Pass project ID to job

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9730: URL: https://github.com/apache/hudi/pull/9730#issuecomment-1722098439 ## CI report: * e998928a33653fb70a7a16e86b141c3077659214 Azure:

[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata

2023-09-15 Thread via GitHub
danny0405 commented on code in PR #9651: URL: https://github.com/apache/hudi/pull/9651#discussion_r1327885439 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java: ## @@ -349,8 +351,8 @@ private static void

[GitHub] [hudi] lokesh-lingarajan-0310 commented on a diff in pull request #9729: [HUDI-6869] Fixing schema evolution docs

2023-09-15 Thread via GitHub
lokesh-lingarajan-0310 commented on code in PR #9729: URL: https://github.com/apache/hudi/pull/9729#discussion_r1327884782 ## website/docs/schema_evolution.md: ## @@ -370,3 +196,180 @@ scala> spark.sql("select rowId, partitionId, preComb, name, versionId, intToLong

[hudi] branch asf-site updated: Fixing schema evolution docs (#9729)

2023-09-15 Thread bhavanisudha
This is an automated email from the ASF dual-hosted git repository. bhavanisudha pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 8f3b215e1e5 Fixing schema evolution

[GitHub] [hudi] bhasudha merged pull request #9729: [HUDI-6869] Fixing schema evolution docs

2023-09-15 Thread via GitHub
bhasudha merged PR #9729: URL: https://github.com/apache/hudi/pull/9729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch master updated: [HUDI-6863] Revert auto-tuning of dedup parallelism (#9722)

2023-09-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new ea8f9258ec1 [HUDI-6863] Revert auto-tuning of

[GitHub] [hudi] yihua merged pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
yihua merged PR #9722: URL: https://github.com/apache/hudi/pull/9722 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] yihua opened a new pull request, #9731: [MINOR] Add tests on combine parallelism

2023-09-15 Thread via GitHub
yihua opened a new pull request, #9731: URL: https://github.com/apache/hudi/pull/9731 ### Change Logs This PR add tests on combine parallelism. This should be landed after #9722. ### Impact No impact on production code logic. ### Risk level none ###

[GitHub] [hudi] yihua commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
yihua commented on PR #9722: URL: https://github.com/apache/hudi/pull/9722#issuecomment-1722092249 CI is green. https://github.com/apache/hudi/assets/2497195/1ef47409-c05f-440c-bda8-b421e705a8ed;> -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] hudi-bot commented on pull request #9730: [HUDI-6870] Pass project ID to job

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9730: URL: https://github.com/apache/hudi/pull/9730#issuecomment-1722086494 ## CI report: * e998928a33653fb70a7a16e86b141c3077659214 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9730: [HUDI-6870] Pass project ID to job

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9730: URL: https://github.com/apache/hudi/pull/9730#issuecomment-1722084098 ## CI report: * e998928a33653fb70a7a16e86b141c3077659214 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] emkornfield commented on pull request #9730: [HUDI-6870] Pass project ID to job

2023-09-15 Thread via GitHub
emkornfield commented on PR #9730: URL: https://github.com/apache/hudi/pull/9730#issuecomment-1722075460 Hmm, need to look into why it can't find the symbol. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[jira] [Updated] (HUDI-6870) [BigQuerySyncTool] Pass target project id when running job.

2023-09-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6870: - Labels: pull-request-available (was: ) > [BigQuerySyncTool] Pass target project id when running

[GitHub] [hudi] emkornfield opened a new pull request, #9730: [HUDI-6870] Pass project ID to job

2023-09-15 Thread via GitHub
emkornfield opened a new pull request, #9730: URL: https://github.com/apache/hudi/pull/9730 ### Change Logs Use the project ID for the table when running the job. ### Impact This will requires users have permissions to run jobs in the target project that they are

[jira] [Created] (HUDI-6870) [BigQuerySyncTool] Pass target project id when running job.

2023-09-15 Thread Micah Kornfield (Jira)
Micah Kornfield created HUDI-6870: - Summary: [BigQuerySyncTool] Pass target project id when running job. Key: HUDI-6870 URL: https://issues.apache.org/jira/browse/HUDI-6870 Project: Apache Hudi

[jira] [Updated] (HUDI-6869) fix schema evol docs to move OOB to first section

2023-09-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6869: - Labels: pull-request-available (was: ) > fix schema evol docs to move OOB to first section >

[GitHub] [hudi] nsivabalan opened a new pull request, #9729: [HUDI-6869] Fixing schema evolution docs

2023-09-15 Thread via GitHub
nsivabalan opened a new pull request, #9729: URL: https://github.com/apache/hudi/pull/9729 ### Change Logs Fixing schema evolution docs. Moved OOB as first section. Moved comprehensive/schema on read to later section. ### Impact Improves usability and brings more

[jira] [Created] (HUDI-6869) fix schema evol docs to move OOB to first section

2023-09-15 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-6869: - Summary: fix schema evol docs to move OOB to first section Key: HUDI-6869 URL: https://issues.apache.org/jira/browse/HUDI-6869 Project: Apache Hudi

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9712: [HUDI-6851] Fixing Spark quick start guide

2023-09-15 Thread via GitHub
nsivabalan commented on code in PR #9712: URL: https://github.com/apache/hudi/pull/9712#discussion_r1327856715 ## website/docs/sql_dml.md: ## @@ -0,0 +1,189 @@ +--- +title: SQL DML +summary: "In this page, we go will cover details on how to modify data with Hudi tables" +toc:

[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9722: URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721840791 ## CI report: * 09a14fbd65818bab270d260b40fe92fa9d5b28b9 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9717: [DNM] Support Spark 3.5.0

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9717: URL: https://github.com/apache/hudi/pull/9717#issuecomment-1721704135 ## CI report: * 9b8fdd2d1b69da528069e364790b53af1d6150af UNKNOWN * a6c34edf0817f4eb3e4a7b25c829c5ed26bd2a43 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9360: [HUDI-6867] Upgrade thrift's version to 0.13.0

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9360: URL: https://github.com/apache/hudi/pull/9360#issuecomment-1721703381 ## CI report: * 6e58695f888ee4c82f7e20ab386e73b9b193fe00 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9717: [DNM] Support Spark 3.5.0

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9717: URL: https://github.com/apache/hudi/pull/9717#issuecomment-1721694048 ## CI report: * 9b8fdd2d1b69da528069e364790b53af1d6150af UNKNOWN * 54b4038e59a32dfee952fd5002f7b58e34d558c0 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9717: [DNM] Support Spark 3.5.0

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9717: URL: https://github.com/apache/hudi/pull/9717#issuecomment-1721645592 ## CI report: * 9b8fdd2d1b69da528069e364790b53af1d6150af UNKNOWN * 54b4038e59a32dfee952fd5002f7b58e34d558c0 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9722: URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721645684 ## CI report: * ea619c6516678384667d4d7a5fe99974409e41e5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9722: URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721636043 ## CI report: * ea619c6516678384667d4d7a5fe99974409e41e5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9717: [DNM] Support Spark 3.5.0

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9717: URL: https://github.com/apache/hudi/pull/9717#issuecomment-1721635912 ## CI report: * 9b8fdd2d1b69da528069e364790b53af1d6150af UNKNOWN * b971f6391abf519883973b17f11fdaa5733c4ee8 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9726: [MINOR] Build failed using master

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9726: URL: https://github.com/apache/hudi/pull/9726#issuecomment-1721626693 ## CI report: * e1c49890b233f64f3560d974f2df3eaa2a5a3c3d Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9651: URL: https://github.com/apache/hudi/pull/9651#issuecomment-1721626451 ## CI report: * bfe38802d7a9a9c5bc677b19ecaefa72d4335553 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9118: [HUDI-2141] Support flink stream write metrics

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9118: URL: https://github.com/apache/hudi/pull/9118#issuecomment-1721625488 ## CI report: * f6d7dd97c73898206da91b17144326a7dbbffae8 UNKNOWN * c62db1fdf94ee2c1f9b9e539f7a4b1bb866beb7e UNKNOWN * a9b387e611bdc9c492a27c6adffe2bf74662be96 Azure:

[GitHub] [hudi] yihua commented on a diff in pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
yihua commented on code in PR #9722: URL: https://github.com/apache/hudi/pull/9722#discussion_r1327598130 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala: ## @@ -1158,7 +1172,9 @@ class TestHoodieSparkSqlWriter { val

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
nsivabalan commented on code in PR #9722: URL: https://github.com/apache/hudi/pull/9722#discussion_r1327590424 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala: ## @@ -1158,7 +1172,9 @@ class TestHoodieSparkSqlWriter { val

[jira] [Updated] (HUDI-6868) Hudi HiveSync doesn't support extracting passwords from credential store

2023-09-15 Thread Kuldeep Kulkarni (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuldeep Kulkarni updated HUDI-6868: --- Description: We have a customer use-case of running PySpark on [Dataproc

[jira] [Created] (HUDI-6868) Hudi HiveSync doesn't support extracting passwords from credential store

2023-09-15 Thread Kuldeep Kulkarni (Jira)
Kuldeep Kulkarni created HUDI-6868: -- Summary: Hudi HiveSync doesn't support extracting passwords from credential store Key: HUDI-6868 URL: https://issues.apache.org/jira/browse/HUDI-6868 Project:

[GitHub] [hudi] yihua commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
yihua commented on PR #9722: URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721593449 > Lets revisit the problems 6802 was tackliing. Main issue it was addressing is, making our shuffle parallelism dynamic and relative to the incoming df's num partitions. So, if someone is

[GitHub] [hudi] Jason-liujc opened a new issue, #9728: [SUPPORT] Hudi Job fails fast even with long concurrent write retries

2023-09-15 Thread via GitHub
Jason-liujc opened a new issue, #9728: URL: https://github.com/apache/hudi/issues/9728 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at

[GitHub] [hudi] Jason-liujc commented on issue #9512: [SUPPORT] No table level lock when using DynamoDB lock provider

2023-09-15 Thread via GitHub
Jason-liujc commented on issue #9512: URL: https://github.com/apache/hudi/issues/9512#issuecomment-1721575819 Thanks! We are using some of the retry parameters to see we can allow all these writers to go through with optimistic retries eventually. This is the hoodie options

[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9722: URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721567233 ## CI report: * ea619c6516678384667d4d7a5fe99974409e41e5 Azure:

[GitHub] [hudi] nsivabalan commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
nsivabalan commented on PR #9722: URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721566580 Lets revisit the problems 6802 was tackliing. Main issue it was addressing is, making our shuffle parallelism dynamic and relative to the incoming df's num partitions. So, if someone is

[GitHub] [hudi] hudi-bot commented on pull request #9360: [HUDI-6867] Upgrade thrift's version to 0.13.0

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9360: URL: https://github.com/apache/hudi/pull/9360#issuecomment-1721566406 ## CI report: * 6e58695f888ee4c82f7e20ab386e73b9b193fe00 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9722: URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721556511 ## CI report: * ea619c6516678384667d4d7a5fe99974409e41e5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-6867) Upgrade thrift's version to 0.13.0

2023-09-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6867: - Labels: pull-request-available (was: ) > Upgrade thrift's version to 0.13.0 >

[GitHub] [hudi] hudi-bot commented on pull request #9360: [HUDI-6867] Upgrade thrift's version to 0.13.0

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9360: URL: https://github.com/apache/hudi/pull/9360#issuecomment-1721555416 ## CI report: * 6e58695f888ee4c82f7e20ab386e73b9b193fe00 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Created] (HUDI-6867) Upgrade thrift's version to 0.13.0

2023-09-15 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-6867: --- Summary: Upgrade thrift's version to 0.13.0 Key: HUDI-6867 URL: https://issues.apache.org/jira/browse/HUDI-6867 Project: Apache Hudi Issue Type: Improvement

[jira] [Updated] (HUDI-6867) Upgrade thrift's version to 0.13.0

2023-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6867: Fix Version/s: 1.0.0 > Upgrade thrift's version to 0.13.0 > -- > >

[GitHub] [hudi] voonhous commented on pull request #9724: [HUDI-6865] Fix InternalSchema schemaId when column is dropped

2023-09-15 Thread via GitHub
voonhous commented on PR #9724: URL: https://github.com/apache/hudi/pull/9724#issuecomment-1721551178 > > @codope IIUC, this should only affect Spark3.2 right? > > @voonhous it should affect all spark 3 versions. Bug is in hudi side and it happens only schema on read is enabled and

[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #9667: [HUDI-6836] Shutting down deltastreamer in tests and shutting down metrics for write client

2023-09-15 Thread via GitHub
the-other-tim-brown commented on code in PR #9667: URL: https://github.com/apache/hudi/pull/9667#discussion_r1327523911 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -1363,6 +1363,10 @@ public void close() { //

[GitHub] [hudi] yihua commented on pull request #9416: [HUDI-6678] Fix the acquisition of clean instants to archive

2023-09-15 Thread via GitHub
yihua commented on PR #9416: URL: https://github.com/apache/hudi/pull/9416#issuecomment-1721527742 Taking a final pass now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[hudi] branch master updated: [MINOR] Close record readers in TestHoodieReaderWriterBase after use during tests (#9504)

2023-09-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new ead5171e2a6 [MINOR] Close record readers in

[GitHub] [hudi] yihua merged pull request #9504: [MINOR] Close record readers in TestHoodieReaderWriterBase after use during tests

2023-09-15 Thread via GitHub
yihua merged PR #9504: URL: https://github.com/apache/hudi/pull/9504 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] Nitish-sati opened a new issue, #9727: Parquet file corruption in hudi

2023-09-15 Thread via GitHub
Nitish-sati opened a new issue, #9727: URL: https://github.com/apache/hudi/issues/9727 I'm currently operating a Spark Streaming job on an EMR cluster, where it retrieves data from an S3 source, performs upsert operations, and then stores it in the Hudi format. Additionally, I'm utilizing

[jira] [Created] (HUDI-6866) When invalidate the table in the spark sql query cache, verify if the hive-async database exists

2023-09-15 Thread Jira
陈磊 created HUDI-6866: Summary: When invalidate the table in the spark sql query cache, verify if the hive-async database exists Key: HUDI-6866 URL: https://issues.apache.org/jira/browse/HUDI-6866 Project: Apache

[jira] (HUDI-6866) When invalidate the table in the spark sql query cache, verify if the hive-async database exists

2023-09-15 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-6866 ] 陈磊 deleted comment on HUDI-6866: -- was (Author: empcl): https://github.com/apache/hudi/pull/9425 > When invalidate the table in the spark sql query cache, verify if the > hive-async database exists >

[jira] [Commented] (HUDI-6866) When invalidate the table in the spark sql query cache, verify if the hive-async database exists

2023-09-15 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-6866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17765686#comment-17765686 ] 陈磊 commented on HUDI-6866: -- https://github.com/apache/hudi/pull/9425 > When invalidate the table in the spark

[GitHub] [hudi] hudi-bot commented on pull request #9726: [MINOR] Build failed using master

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9726: URL: https://github.com/apache/hudi/pull/9726#issuecomment-1721419393 ## CI report: * 1804bbe036fb68361de16b574e5caaf58334ecf9 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9118: [HUDI-2141] Support flink stream write metrics

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9118: URL: https://github.com/apache/hudi/pull/9118#issuecomment-1721417392 ## CI report: * f6d7dd97c73898206da91b17144326a7dbbffae8 UNKNOWN * c62db1fdf94ee2c1f9b9e539f7a4b1bb866beb7e UNKNOWN * 9a234ef222aaced75692ea0f2c828aad9ef339c5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9726: [MINOR] Build failed using master

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9726: URL: https://github.com/apache/hudi/pull/9726#issuecomment-1721404027 ## CI report: * 1804bbe036fb68361de16b574e5caaf58334ecf9 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9118: [HUDI-2141] Support flink stream write metrics

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9118: URL: https://github.com/apache/hudi/pull/9118#issuecomment-1721402370 ## CI report: * f6d7dd97c73898206da91b17144326a7dbbffae8 UNKNOWN * c62db1fdf94ee2c1f9b9e539f7a4b1bb866beb7e UNKNOWN * 9a234ef222aaced75692ea0f2c828aad9ef339c5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9651: URL: https://github.com/apache/hudi/pull/9651#issuecomment-1721389341 ## CI report: * bfe38802d7a9a9c5bc677b19ecaefa72d4335553 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9625: [MINOR] Fix default config values if not specified in MultipleSparkJobExecutionStrategy

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9625: URL: https://github.com/apache/hudi/pull/9625#issuecomment-1721389120 ## CI report: * 618b28e2d166f92b214f0f5d992d0d9d6415d8e9 Azure:

[GitHub] [hudi] codope commented on pull request #9724: [HUDI-6865] Fix InternalSchema schemaId when column is dropped

2023-09-15 Thread via GitHub
codope commented on PR #9724: URL: https://github.com/apache/hudi/pull/9724#issuecomment-1721304662 > @codope IIUC, this should only affect Spark3.2 right? @voonhous it should affect all spark 3 versions. Bug is in hudi side and it happens only schema on read is enabled and a

[GitHub] [hudi] Forus0322 commented on issue #9725: Build failed using master,

2023-09-15 Thread via GitHub
Forus0322 commented on issue #9725: URL: https://github.com/apache/hudi/issues/9725#issuecomment-1721298540 @codope hadoop 2.10.x references the org.codehaus.jackson:jackson-core-asl dependency package, but hadoop 3.3.0 does not have this dependency package. Spark 3.1 Profile uses

[GitHub] [hudi] hudi-bot commented on pull request #9585: [HUDI-6809] Optimizing the judgment of generating clustering plans

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9585: URL: https://github.com/apache/hudi/pull/9585#issuecomment-1721294375 ## CI report: * 3cef8796dbd3f5fe66085f0f207b487b4aa22b21 UNKNOWN * 80f21a38fa4ad5ad7c6e3ec503414fb2a0e063f8 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9564: [HUDI-6712] Add Parquet file metadata loader

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9564: URL: https://github.com/apache/hudi/pull/9564#issuecomment-1721227256 ## CI report: * be94a005ed6e1306c656c859cd42d5378958e414 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9726: [MINOR] Build failed using master

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9726: URL: https://github.com/apache/hudi/pull/9726#issuecomment-1721214217 ## CI report: * 1804bbe036fb68361de16b574e5caaf58334ecf9 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9651: URL: https://github.com/apache/hudi/pull/9651#issuecomment-1721213785 ## CI report: * bfe38802d7a9a9c5bc677b19ecaefa72d4335553 Azure:

[GitHub] [hudi] bhasudha commented on a diff in pull request #9712: [HUDI-6851] Fixing Spark quick start guide

2023-09-15 Thread via GitHub
bhasudha commented on code in PR #9712: URL: https://github.com/apache/hudi/pull/9712#discussion_r1327198462 ## website/docs/quick-start-guide.md: ## @@ -778,18 +724,20 @@ val updates = convertToStringList(dataGen.generateUpdates(10)) val df =

[GitHub] [hudi] hudi-bot commented on pull request #9724: [HUDI-6865] Fix InternalSchema schemaId when column is dropped

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9724: URL: https://github.com/apache/hudi/pull/9724#issuecomment-1721132032 ## CI report: * 6aa3d8ee21ebc3f62796128ed2c944f23b24db3b Azure:

[GitHub] [hudi] Forus0322 commented on a diff in pull request #9726: [MINOR] Build failed using master

2023-09-15 Thread via GitHub
Forus0322 commented on code in PR #9726: URL: https://github.com/apache/hudi/pull/9726#discussion_r1327110069 ## hudi-common/pom.xml: ## @@ -286,5 +286,12 @@ disruptor ${disruptor.version} + + Review Comment: I just tried it, this problem will occur

[GitHub] [hudi] hudi-bot commented on pull request #9585: [HUDI-6809] Optimizing the judgment of generating clustering plans

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9585: URL: https://github.com/apache/hudi/pull/9585#issuecomment-1721073198 ## CI report: * 96498146004c1862ac8d3fe1c734696d461c8bf8 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9625: [MINOR] Fix default config values if not specified in MultipleSparkJobExecutionStrategy

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9625: URL: https://github.com/apache/hudi/pull/9625#issuecomment-1721073475 ## CI report: * 0847642fd60ed2147353be24e5a05b3eb3e42e14 Azure:

[GitHub] [hudi] voonhous commented on pull request #9724: [HUDI-6865] Fix InternalSchema schemaId when column is dropped

2023-09-15 Thread via GitHub
voonhous commented on PR #9724: URL: https://github.com/apache/hudi/pull/9724#issuecomment-1721068436 @codope IIUC, this should only affect Spark3.2 right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] hudi-bot commented on pull request #9625: [MINOR] Fix default config values if not specified in MultipleSparkJobExecutionStrategy

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9625: URL: https://github.com/apache/hudi/pull/9625#issuecomment-1721063147 ## CI report: * 0847642fd60ed2147353be24e5a05b3eb3e42e14 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9585: [HUDI-6809] Optimizing the judgment of generating clustering plans

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9585: URL: https://github.com/apache/hudi/pull/9585#issuecomment-1721062904 ## CI report: * 96498146004c1862ac8d3fe1c734696d461c8bf8 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9726: [MINOR] Build failed using master

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9726: URL: https://github.com/apache/hudi/pull/9726#issuecomment-1721053041 ## CI report: * 1804bbe036fb68361de16b574e5caaf58334ecf9 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9651: URL: https://github.com/apache/hudi/pull/9651#issuecomment-1721052739 ## CI report: * 026b65991bcb167f2a361c47fa87c29bc329ef9a Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9585: [HUDI-6809] Optimizing the judgment of generating clustering plans

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9585: URL: https://github.com/apache/hudi/pull/9585#issuecomment-1721052463 ## CI report: * 96498146004c1862ac8d3fe1c734696d461c8bf8 Azure:

[GitHub] [hudi] Forus0322 commented on a diff in pull request #9726: [MINOR] Build failed using master

2023-09-15 Thread via GitHub
Forus0322 commented on code in PR #9726: URL: https://github.com/apache/hudi/pull/9726#discussion_r1327110069 ## hudi-common/pom.xml: ## @@ -286,5 +286,12 @@ disruptor ${disruptor.version} + + Review Comment: I just tried it, this problem will occur

[GitHub] [hudi] codope commented on a diff in pull request #9726: [MINOR] Build failed using master

2023-09-15 Thread via GitHub
codope commented on code in PR #9726: URL: https://github.com/apache/hudi/pull/9726#discussion_r1327084588 ## hudi-common/pom.xml: ## @@ -286,5 +286,12 @@ disruptor ${disruptor.version} + + Review Comment: ``` mvn clean package install -DskipTests

[GitHub] [hudi] hudi-bot commented on pull request #9726: [MINOR] Build failed using master

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9726: URL: https://github.com/apache/hudi/pull/9726#issuecomment-1720995436 ## CI report: * 1804bbe036fb68361de16b574e5caaf58334ecf9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9651: URL: https://github.com/apache/hudi/pull/9651#issuecomment-1720995078 ## CI report: * c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9360: [MINOR] Upgrade thrift's version to 0.13.0

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9360: URL: https://github.com/apache/hudi/pull/9360#issuecomment-1720994247 ## CI report: * 6e58695f888ee4c82f7e20ab386e73b9b193fe00 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9724: [HUDI-6865] Fix InternalSchema schemaId when column is dropped

2023-09-15 Thread via GitHub
hudi-bot commented on PR #9724: URL: https://github.com/apache/hudi/pull/9724#issuecomment-1720983007 ## CI report: * 6aa3d8ee21ebc3f62796128ed2c944f23b24db3b Azure:

[GitHub] [hudi] codope commented on issue #9725: Build failed using master,

2023-09-15 Thread via GitHub
codope commented on issue #9725: URL: https://github.com/apache/hudi/issues/9725#issuecomment-1720974036 Thanks I am checking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] codope commented on a diff in pull request #9726: [MINOR] Build failed using master

2023-09-15 Thread via GitHub
codope commented on code in PR #9726: URL: https://github.com/apache/hudi/pull/9726#discussion_r1327050080 ## hudi-common/pom.xml: ## @@ -286,5 +286,12 @@ disruptor ${disruptor.version} + + Review Comment: Shouldn't this already be included? Perhaps

[GitHub] [hudi] Forus0322 commented on issue #9725: Build failed using master,

2023-09-15 Thread via GitHub
Forus0322 commented on issue #9725: URL: https://github.com/apache/hudi/issues/9725#issuecomment-1720969402 @codope `mvn clean package install -DskipTests -Dspark3.3 -Dscala-2.12 -Dhadoop.version=3.3.0 -Dhive.version=3.1.2 -Phudi-platform-service -Dcheckstyle.skip=true -Drat.skip=true

[GitHub] [hudi] Forus0322 opened a new pull request, #9726: [Hudi-9725] Build failed using master

2023-09-15 Thread via GitHub
Forus0322 opened a new pull request, #9726: URL: https://github.com/apache/hudi/pull/9726 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ Add jackson dependencies. ### Impact _Describe any public API or

  1   2   >