[jira] [Updated] (HUDI-6268) Make HoodieFlinkCompactor could read parameter by specifying options which like --props and --hoodie-conf

2023-05-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6268: - Labels: pull-request-available (was: ) > Make HoodieFlinkCompactor could read parameter by specif

[GitHub] [hudi] eric9204 opened a new pull request, #8815: [HUDI-6268]Make HoodieFlinkCompactor could read parameter by specifying options …

2023-05-25 Thread via GitHub
eric9204 opened a new pull request, #8815: URL: https://github.com/apache/hudi/pull/8815 …which like --props and --hoodie-conf. ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ add options like `--props` and `--hoodie-conf

[GitHub] [hudi] codope merged pull request #8303: [HUDI-5998] Speed up reads from bootstrapped tables in spark

2023-05-25 Thread via GitHub
codope merged PR #8303: URL: https://github.com/apache/hudi/pull/8303 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.or

[hudi] branch master updated: [HUDI-5998] Speed up reads from bootstrapped tables in spark (#8303)

2023-05-25 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 59786113fae [HUDI-5998] Speed up reads from bootst

[GitHub] [hudi] hudi-bot commented on pull request #8814: [MINOR] Updating DOAP file for 0.13.1

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8814: URL: https://github.com/apache/hudi/pull/8814#issuecomment-1563877865 ## CI report: * a268552102f7743550015806bfacf22bf5338d8c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1738

[GitHub] [hudi] eyjian closed issue #8757: [SUPPORT] How to get a row of a primary key?

2023-05-25 Thread via GitHub
eyjian closed issue #8757: [SUPPORT] How to get a row of a primary key? URL: https://github.com/apache/hudi/issues/8757 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

[jira] [Created] (HUDI-6268) Make HoodieFlinkCompactor could read parameter by specifying options which like --props and --hoodie-conf

2023-05-25 Thread eric (Jira)
eric created HUDI-6268: -- Summary: Make HoodieFlinkCompactor could read parameter by specifying options which like --props and --hoodie-conf Key: HUDI-6268 URL: https://issues.apache.org/jira/browse/HUDI-6268 Pro

[GitHub] [hudi] hudi-bot commented on pull request #8814: [MINOR] Updating DOAP file for 0.13.1

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8814: URL: https://github.com/apache/hudi/pull/8814#issuecomment-1563871358 ## CI report: * a268552102f7743550015806bfacf22bf5338d8c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #8604: [HUDI-6151] Rollback previously applied commits to MDT when operations are retried.

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8604: URL: https://github.com/apache/hudi/pull/8604#issuecomment-1563870669 ## CI report: * 7612c1e37dca940b98910d81f849a49740f81cc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1738

[GitHub] [hudi] hudi-bot commented on pull request #8452: [HUDI-6077] Add more partition push down filters

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8452: URL: https://github.com/apache/hudi/pull/8452#issuecomment-1563870400 ## CI report: * 8082df232089396b2a9f9be2b915e51b3645f172 UNKNOWN * 70072cdcfd491bc98e76eac3c75b63f746fa2d6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[hudi] branch master updated: [HUDI-6041] add `options` input to Bootstrap Procedure for passing hudi properties (#8387)

2023-05-25 Thread vbalaji
This is an automated email from the ASF dual-hosted git repository. vbalaji pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 1725b0915ad [HUDI-6041] add `options` input to Bo

[GitHub] [hudi] bvaradar merged pull request #8387: [HUDI-6041] add `options` input to Bootstrap Procedure for passing hudi properties

2023-05-25 Thread via GitHub
bvaradar merged PR #8387: URL: https://github.com/apache/hudi/pull/8387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[GitHub] [hudi] boneanxs commented on pull request #8076: [HUDI-5884] Support bulk_insert for insert_overwrite and insert_overwrite_table

2023-05-25 Thread via GitHub
boneanxs commented on PR #8076: URL: https://github.com/apache/hudi/pull/8076#issuecomment-1563854865 Hey, @codope, all comments are resolved, could you pls review it again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hudi] danny0405 commented on a diff in pull request #8807: [HUDI-6259] support configuration hot update for deltastreamer continuous mode

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8807: URL: https://github.com/apache/hudi/pull/8807#discussion_r1206267911 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ConfigurationHotUpdateStrategy.java: ## @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] danny0405 commented on a diff in pull request #8807: [HUDI-6259] support configuration hot update for deltastreamer continuous mode

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8807: URL: https://github.com/apache/hudi/pull/8807#discussion_r1206267533 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ConfigurationHotUpdateStrategy.java: ## @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] zhangyue19921010 opened a new pull request, #8814: [MINOR] Updating DOAP file for 0.13.1

2023-05-25 Thread via GitHub
zhangyue19921010 opened a new pull request, #8814: URL: https://github.com/apache/hudi/pull/8814 ### Change Logs Updating DOAP file for 0.12.3 ### Impact Updating DOAP file for 0.12.3 ### Risk level (write none, low medium or high below) none ### Docu

[GitHub] [hudi] hudi-bot commented on pull request #8811: [Minor] hotfix in HoodieSparkEngineContext with delete log

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8811: URL: https://github.com/apache/hudi/pull/8811#issuecomment-1563834279 ## CI report: * 1b21820a618da396e4ba70fece09bb28b3d8fd39 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=173

[GitHub] [hudi] danny0405 commented on a diff in pull request #8668: [HUDI-3639] Add Proper Incremental Records FIltering support into Hudi's custom RDD

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8668: URL: https://github.com/apache/hudi/pull/8668#discussion_r1206256557 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDD.scala: ## @@ -116,7 +125,45 @@ class HoodieMergeOnReadRDD(@transient sc: S

[GitHub] [hudi] codope commented on pull request #8303: [HUDI-5998] Speed up reads from bootstrapped tables in spark

2023-05-25 Thread via GitHub
codope commented on PR #8303: URL: https://github.com/apache/hudi/pull/8303#issuecomment-1563829690 @bvaradar The changes looks good to me. Can you take another pass? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hudi] hudi-bot commented on pull request #8813: [DO NOT MERGE][HUDI-6198][0.13.0] Testing Spark 3.4.0 Upgrade

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8813: URL: https://github.com/apache/hudi/pull/8813#issuecomment-1563829650 ## CI report: * 771561634feebeb4896c8dd88862292f6251bde7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #8811: [Minor] hotfix in HoodieSparkEngineContext with delete log

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8811: URL: https://github.com/apache/hudi/pull/8811#issuecomment-1563829592 ## CI report: * 1b21820a618da396e4ba70fece09bb28b3d8fd39 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1738

[jira] [Updated] (HUDI-6257) upgrade table version with hive style path will not check default path

2023-05-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6257: - Labels: pull-request-available (was: ) > upgrade table version with hive style path will not chec

[GitHub] [hudi] danny0405 commented on a diff in pull request #8794: [HUDI-6257] fix upgrade table version check partition path with hive …

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8794: URL: https://github.com/apache/hudi/pull/8794#discussion_r1206254627 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FourToFiveUpgradeHandler.java: ## @@ -71,4 +83,26 @@ public Map upgrade(HoodieWriteConfig

[hudi] branch master updated: [HUDI-5517][FOLLOW-UP] Refine API names and ensure time travel won't affect by stateTransitionTime (#8762)

2023-05-25 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 83ea5016478 [HUDI-5517][FOLLOW-UP] Refine API n

[GitHub] [hudi] danny0405 commented on pull request #8716: [HUDI-6226] Support parquet native bloom filters

2023-05-25 Thread via GitHub
danny0405 commented on PR #8716: URL: https://github.com/apache/hudi/pull/8716#issuecomment-1563823699 Thanks for the sharing, I think the Databricks BloomFilter index mainly serves as query optimization purposes right? Do they also use this to accelate the data skipping during data ingesti

[GitHub] [hudi] danny0405 commented on pull request #8792: [HUDI-6256] Fix the data table archiving and MDT cleaning config conf…

2023-05-25 Thread via GitHub
danny0405 commented on PR #8792: URL: https://github.com/apache/hudi/pull/8792#issuecomment-1563821496 @flashJd Can you check the test failure for flink: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=17368&view=logs&j=600e7de6-e133-5e69-e615-50ee129b3c08&t=bb

[GitHub] [hudi] xuzifu666 commented on a diff in pull request #8811: [Minor] hotfix in HoodieSparkEngineContext with delete log

2023-05-25 Thread via GitHub
xuzifu666 commented on code in PR #8811: URL: https://github.com/apache/hudi/pull/8811#discussion_r1206245965 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java: ## @@ -62,8 +62,7 @@ */ @ThreadSafe public class HoodieSp

[GitHub] [hudi] danny0405 commented on a diff in pull request #8811: [Minor] hotfix in HoodieSparkEngineContext with delete log

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8811: URL: https://github.com/apache/hudi/pull/8811#discussion_r1206245192 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java: ## @@ -62,8 +62,7 @@ */ @ThreadSafe public class HoodieSp

[GitHub] [hudi] ad1happy2go commented on issue #8791: [SUPPORT] ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-05-25 Thread via GitHub
ad1happy2go commented on issue #8791: URL: https://github.com/apache/hudi/issues/8791#issuecomment-1563809208 @lucienoz Were you able to resolve this issue? If yes please share the root cause for the same. -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [hudi] rahil-c opened a new pull request, #8813: [DO NOT MERGE][HUDI-6198][0.13.0] Testing Spark 3.4.0 Upgrade

2023-05-25 Thread via GitHub
rahil-c opened a new pull request, #8813: URL: https://github.com/apache/hudi/pull/8813 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance

[GitHub] [hudi] hudi-bot commented on pull request #8812: [HUDI-6267] init deletedFileResult with false to avoid NPE

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8812: URL: https://github.com/apache/hudi/pull/8812#issuecomment-1563797467 ## CI report: * c6e2e697c9cf1df66e5b5171e7372ff0ba5e6446 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1738

[GitHub] [hudi] hudi-bot commented on pull request #8811: [Minor] hotfix in HoodieSparkEngineContext with delete log

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8811: URL: https://github.com/apache/hudi/pull/8811#issuecomment-1563797449 ## CI report: * 1b21820a618da396e4ba70fece09bb28b3d8fd39 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1738

[GitHub] [hudi] hudi-bot commented on pull request #8811: [Minor] hotfix in HoodieSparkEngineContext with delete log

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8811: URL: https://github.com/apache/hudi/pull/8811#issuecomment-1563793550 ## CI report: * 1b21820a618da396e4ba70fece09bb28b3d8fd39 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #8812: [HUDI-6267] init deletedFileResult with false to avoid NPE

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8812: URL: https://github.com/apache/hudi/pull/8812#issuecomment-1563793571 ## CI report: * c6e2e697c9cf1df66e5b5171e7372ff0ba5e6446 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-6267) CleanActionExecutor throws NPE when encounter IOException

2023-05-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6267: - Labels: pull-request-available (was: ) > CleanActionExecutor throws NPE when encounter IOExceptio

[GitHub] [hudi] dengweisysu opened a new pull request, #8812: [HUDI-6267] init deletedFileResult with false to avoid NPE

2023-05-25 Thread via GitHub
dengweisysu opened a new pull request, #8812: URL: https://github.com/apache/hudi/pull/8812 ### Change Logs 1)init deletedFileResult with false 2) log exception detail ### Impact when encounter IOException , deleteFile will finish normally, but may result files left

[GitHub] [hudi] xuzifu666 opened a new pull request, #8811: [Minor] hotfix in HoodieSparkEngineContext with delete log

2023-05-25 Thread via GitHub
xuzifu666 opened a new pull request, #8811: URL: https://github.com/apache/hudi/pull/8811 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact no impact,only minor hotfix _Describe any public API or user-facing f

[jira] [Created] (HUDI-6267) CleanActionExecutor throws NPE when encounter IOException

2023-05-25 Thread wesleydeng_nouse (Jira)
wesleydeng_nouse created HUDI-6267: -- Summary: CleanActionExecutor throws NPE when encounter IOException Key: HUDI-6267 URL: https://issues.apache.org/jira/browse/HUDI-6267 Project: Apache Hudi

[GitHub] [hudi] king5holiday commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeE

2023-05-25 Thread via GitHub
king5holiday commented on issue #8810: URL: https://github.com/apache/hudi/issues/8810#issuecomment-1563768961 > Flink @danny0405 Thank you for your replay! yes, I wrote data to hudi by flink and the version of hudi is 0.11.1 -- This is an automated message from the Apache Git Serv

[GitHub] [hudi] king5holiday commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeE

2023-05-25 Thread via GitHub
king5holiday commented on issue #8810: URL: https://github.com/apache/hudi/issues/8810#issuecomment-1563768292 > Thank you for your replay! yes, I wrote data to hudi by flink and the version of hudi is 0.11.1 -- This is an automated message from the Apache Git Service. To respond to t

[GitHub] [hudi] king5holiday commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeE

2023-05-25 Thread via GitHub
king5holiday commented on issue #8810: URL: https://github.com/apache/hudi/issues/8810#issuecomment-1563768270 > Thank you for your replay! yes, I wrote data to hudi by flink and the version of hudi is 0.11.1 -- This is an automated message from the Apache Git Service. To respond to t

[GitHub] [hudi] hudi-bot commented on pull request #8604: [HUDI-6151] Rollback previously applied commits to MDT when operations are retried.

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8604: URL: https://github.com/apache/hudi/pull/8604#issuecomment-1563767387 ## CI report: * 0901a75c382c08ad0eb95e2173a3adc331de859e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1732

[GitHub] [hudi] hudi-bot commented on pull request #8604: [HUDI-6151] Rollback previously applied commits to MDT when operations are retried.

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8604: URL: https://github.com/apache/hudi/pull/8604#issuecomment-1563763868 ## CI report: * 0901a75c382c08ad0eb95e2173a3adc331de859e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1732

[GitHub] [hudi] danny0405 commented on a diff in pull request #8604: [HUDI-6151] Rollback previously applied commits to MDT when operations are retried.

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8604: URL: https://github.com/apache/hudi/pull/8604#discussion_r1206201500 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java: ## @@ -161,27 +161,28 @@ protected void commit(String

[GitHub] [hudi] danny0405 commented on a diff in pull request #8787: [HUDI-6254] Allow using absolute path in ManifestFileWriter

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8787: URL: https://github.com/apache/hudi/pull/8787#discussion_r1206190413 ## hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncTool.java: ## @@ -96,7 +96,7 @@ private void syncCoWTable(HoodieBigQuerySyncClient bqSyncClient) {

[GitHub] [hudi] danny0405 commented on a diff in pull request #8797: [HUDI-6260] Fix the MDT compaction reader with the instant range filt…

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8797: URL: https://github.com/apache/hudi/pull/8797#discussion_r1206186838 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java: ## @@ -247,6 +256,17 @@ public List compact(HoodieCompactionHa

[GitHub] [hudi] loukey-lj commented on pull request #6612: [HUDI-4790][RFC-68] a more effective HoodieMergeHandler for COW table with parquet

2023-05-25 Thread via GitHub
loukey-lj commented on PR #6612: URL: https://github.com/apache/hudi/pull/6612#issuecomment-1563733359 > Is this RFC only valid for SQL update scenarios, because it can parse out which columns have been updated from SQL statement. But in other scenarios, such as the "mysql -> debezium -> ka

[GitHub] [hudi] danny0405 commented on a diff in pull request #8797: [HUDI-6260] Fix the MDT compaction reader with the instant range filt…

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8797: URL: https://github.com/apache/hudi/pull/8797#discussion_r1206169642 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java: ## @@ -201,6 +209,7 @@ public List compact(HoodieCompactionHan

[GitHub] [hudi] danny0405 commented on a diff in pull request #8797: [HUDI-6260] Fix the MDT compaction reader with the instant range filt…

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8797: URL: https://github.com/apache/hudi/pull/8797#discussion_r1206169146 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/InstantRange.java: ## @@ -174,6 +195,11 @@ public Builder nullableBoundary(boolean nullable) { retur

[GitHub] [hudi] danny0405 commented on a diff in pull request #8782: [HUDI-6201] use concurrent map when possible in filesystemview

2023-05-25 Thread via GitHub
danny0405 commented on code in PR #8782: URL: https://github.com/apache/hudi/pull/8782#discussion_r1206162895 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieTableFileSystemView.java: ## @@ -199,7 +201,7 @@ protected boolean isPendingCompactionScheduledForF

[GitHub] [hudi] king5holiday opened a new issue, #8810: when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException:

2023-05-25 Thread via GitHub
king5holiday opened a new issue, #8810: URL: https://github.com/apache/hudi/issues/8810 so i want to know why there will be some parquet files which the file name start with dot(.) when I write data to hudi? And how to filter these files when I read hudi by spark? Thank you very much! --

[GitHub] [hudi] boneanxs closed pull request #8669: [HUDI-5362] Rebase IncrementalRelation over HoodieBaseRelation

2023-05-25 Thread via GitHub
boneanxs closed pull request #8669: [HUDI-5362] Rebase IncrementalRelation over HoodieBaseRelation URL: https://github.com/apache/hudi/pull/8669 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [hudi] boneanxs commented on pull request #8669: [HUDI-5362] Rebase IncrementalRelation over HoodieBaseRelation

2023-05-25 Thread via GitHub
boneanxs commented on PR #8669: URL: https://github.com/apache/hudi/pull/8669#issuecomment-1563714652 > I guess my PR is not fully reviewed and it's left behind. Should I apply the difference on https://github.com/apache/hudi/pull/6045 and close this one? Yea, sure, let me close it.

[GitHub] [hudi] xushiyan commented on a diff in pull request #8445: [HUDI-3088] Use Spark 3.2 as default Spark version

2023-05-25 Thread via GitHub
xushiyan commented on code in PR #8445: URL: https://github.com/apache/hudi/pull/8445#discussion_r1206146854 ## hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java: ## @@ -1946,8 +1947,8 @@ public void testJsonKafkaDFSSource() throws

[jira] [Created] (HUDI-6266) Improve hudi-utilities tests with KafkaTestUtils to allow more partitions

2023-05-25 Thread Raymond Xu (Jira)
Raymond Xu created HUDI-6266: Summary: Improve hudi-utilities tests with KafkaTestUtils to allow more partitions Key: HUDI-6266 URL: https://issues.apache.org/jira/browse/HUDI-6266 Project: Apache Hudi

[jira] [Closed] (HUDI-3088) Make Spark 3 the default profile for build and test

2023-05-25 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-3088. Resolution: Fixed > Make Spark 3 the default profile for build and test > --

[hudi] branch master updated (516c3d59404 -> df64d647fa3)

2023-05-25 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 516c3d59404 [HUDI-3088] Use Spark 3.2 as default Spark version (#8445) add df64d647fa3 [HUDI-1517] create marker

[GitHub] [hudi] hudi-bot commented on pull request #8808: [HUDI-6264] Assert that bootstrap base path is different than path

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8808: URL: https://github.com/apache/hudi/pull/8808#issuecomment-1563691864 ## CI report: * f1a6432eb38c006174e7caada19cba73a6c008f9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1738

[GitHub] [hudi] eyjian commented on issue #8757: [SUPPORT] How to get a row of a primary key?

2023-05-25 Thread via GitHub
eyjian commented on issue #8757: URL: https://github.com/apache/hudi/issues/8757#issuecomment-1563690431 > PartialUpdateAvroPayload is only supported in 0.13.0. > > So Can you try with that version and let us know if you still get the issue. I confirmed its working as expected as past

[hudi] branch master updated (c859ea4cd23 -> 516c3d59404)

2023-05-25 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from c859ea4cd23 Revert "[MINOR] Check the return value from delete during rollback and finalize to ensure the files actua

[GitHub] [hudi] xushiyan merged pull request #8445: [HUDI-3088] Use Spark 3.2 as default Spark version

2023-05-25 Thread via GitHub
xushiyan merged PR #8445: URL: https://github.com/apache/hudi/pull/8445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

[GitHub] [hudi] BruceKellan commented on a diff in pull request #7561: [HUDI-5477] Optimize timeline loading in Hudi sync client

2023-05-25 Thread via GitHub
BruceKellan commented on code in PR #7561: URL: https://github.com/apache/hudi/pull/7561#discussion_r1206130102 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java: ## @@ -210,11 +210,30 @@ public static HoodieDefaultTimeline getTimeline(Hoodie

[GitHub] [hudi] yihua commented on a diff in pull request #8797: [HUDI-6260] Fix the MDT compaction reader with the instant range filt…

2023-05-25 Thread via GitHub
yihua commented on code in PR #8797: URL: https://github.com/apache/hudi/pull/8797#discussion_r1206094130 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/InstantRange.java: ## @@ -174,6 +195,11 @@ public Builder nullableBoundary(boolean nullable) { return th

[GitHub] [hudi] yihua commented on a diff in pull request #8797: [HUDI-6260] Fix the MDT compaction reader with the instant range filt…

2023-05-25 Thread via GitHub
yihua commented on code in PR #8797: URL: https://github.com/apache/hudi/pull/8797#discussion_r1206088234 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java: ## @@ -201,6 +209,7 @@ public List compact(HoodieCompactionHandler

[GitHub] [hudi] hudi-bot commented on pull request #4913: [HUDI-1517] create marker file for every log file

2023-05-25 Thread via GitHub
hudi-bot commented on PR #4913: URL: https://github.com/apache/hudi/pull/4913#issuecomment-1563624940 ## CI report: * 8294ef2e36bbc959f3bb17b804e83d57eea05196 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1737

[jira] [Updated] (HUDI-6263) Update hoodie.properties will cause reader failed: hoodie.properties: No such file or directory!

2023-05-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6263: - Labels: pull-request-available (was: ) > Update hoodie.properties will cause reader failed: hoodi

[GitHub] [hudi] nsivabalan commented on pull request #8804: [HUDI-6263] Fix update HoodieTableConfig cause reader exception

2023-05-25 Thread via GitHub
nsivabalan commented on PR #8804: URL: https://github.com/apache/hudi/pull/8804#issuecomment-1563616330 CC @nbalajee @prashantwason @suryaprasanna : you folks had some fix on this end right? -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [hudi] hudi-bot commented on pull request #8808: [HUDI-6264] Assert that bootstrap base path is different than path

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8808: URL: https://github.com/apache/hudi/pull/8808#issuecomment-1563592645 ## CI report: * e3e35985ecfe1c07aeca403476a9ded07eebef08 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1737

[GitHub] [hudi] hudi-bot commented on pull request #8808: [HUDI-6264] Assert that bootstrap base path is different than path

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8808: URL: https://github.com/apache/hudi/pull/8808#issuecomment-1563586493 ## CI report: * e3e35985ecfe1c07aeca403476a9ded07eebef08 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1737

[GitHub] [hudi] hudi-bot commented on pull request #8809: [HUDI-4944] use mor solution to uri filenotfound problem

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8809: URL: https://github.com/apache/hudi/pull/8809#issuecomment-1563578976 ## CI report: * 10519b1e65cc4a6b6e397bcf206235397b5c213a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1737

[jira] [Updated] (HUDI-6265) Fix timeline check for meta sync

2023-05-25 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6265: Priority: Critical (was: Major) > Fix timeline check for meta sync > > >

[jira] [Updated] (HUDI-6265) Fix timeline check for meta sync

2023-05-25 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6265: Description: [https://github.com/apache/hudi/pull/7561/files#r1203654251] The following case where the time

[jira] [Updated] (HUDI-6265) Fix timeline check for meta sync

2023-05-25 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6265: Affects Version/s: 0.13.0 > Fix timeline check for meta sync > > >

[GitHub] [hudi] yihua commented on a diff in pull request #7561: [HUDI-5477] Optimize timeline loading in Hudi sync client

2023-05-25 Thread via GitHub
yihua commented on code in PR #7561: URL: https://github.com/apache/hudi/pull/7561#discussion_r1206050514 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java: ## @@ -210,11 +210,30 @@ public static HoodieDefaultTimeline getTimeline(HoodieTableM

[jira] [Created] (HUDI-6265) Fix timeline check for meta sync

2023-05-25 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-6265: --- Summary: Fix timeline check for meta sync Key: HUDI-6265 URL: https://issues.apache.org/jira/browse/HUDI-6265 Project: Apache Hudi Issue Type: Improvement

[jira] [Assigned] (HUDI-6265) Fix timeline check for meta sync

2023-05-25 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-6265: --- Assignee: Ethan Guo > Fix timeline check for meta sync > > >

[jira] [Updated] (HUDI-6265) Fix timeline check for meta sync

2023-05-25 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6265: Issue Type: Bug (was: Improvement) > Fix timeline check for meta sync > >

[jira] [Updated] (HUDI-6265) Fix timeline check for meta sync

2023-05-25 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6265: Fix Version/s: 0.14.0 > Fix timeline check for meta sync > > >

[GitHub] [hudi] hudi-bot commented on pull request #8808: [HUDI-6264] Assert that bootstrap base path is different than path

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8808: URL: https://github.com/apache/hudi/pull/8808#issuecomment-1563518771 ## CI report: * e3e35985ecfe1c07aeca403476a9ded07eebef08 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1737

[GitHub] [hudi] hudi-bot commented on pull request #8445: [HUDI-3088] Use Spark 3.2 as default Spark version

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8445: URL: https://github.com/apache/hudi/pull/8445#issuecomment-1563517792 ## CI report: * 1f9f158675ba301312206710df2fad27982bc0b3 UNKNOWN * 8bc8b9c46b1c1e2578df748fc91039cf781dcd11 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #8574: [HUDI-6139] Add support for Transformer schema validation in DeltaStreamer

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8574: URL: https://github.com/apache/hudi/pull/8574#issuecomment-1563410797 ## CI report: * dacba722974aa32f506626c106d90fa86d22cd23 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1737

[jira] [Updated] (HUDI-6259) deltastreamer support configuration hot update

2023-05-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6259: - Labels: pull-request-available (was: ) > deltastreamer support configuration hot update > ---

[GitHub] [hudi] hudi-bot commented on pull request #8807: [HUDI-6259] support configuration hot update for deltastreamer continuous mode

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8807: URL: https://github.com/apache/hudi/pull/8807#issuecomment-1563337596 ## CI report: * f887f9ccc721c937f212fa4df30021201c82cf37 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1737

[GitHub] [hudi] mzheng-plaid commented on issue #8715: [SUPPORT] High number of REST.GET.BUCKET when writing over a hudi table

2023-05-25 Thread via GitHub
mzheng-plaid commented on issue #8715: URL: https://github.com/apache/hudi/issues/8715#issuecomment-1563337286 > #7404 Ah, thats interesting - does this mean we need to set https://hudi.apache.org/docs/0.12.3/configurations/#hoodiebootstrapindexclass to https://github.com/apache/hud

[GitHub] [hudi] xushiyan commented on a diff in pull request #4913: [HUDI-1517] create marker file for every log file

2023-05-25 Thread via GitHub
xushiyan commented on code in PR #4913: URL: https://github.com/apache/hudi/pull/4913#discussion_r1205870328 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkAppendHandle.java: ## @@ -60,17 +63,28 @@ public FlinkAppendHandle( Iterator> recordItr,

[GitHub] [hudi] xushiyan commented on a diff in pull request #4913: [HUDI-1517] create marker file for every log file

2023-05-25 Thread via GitHub
xushiyan commented on code in PR #4913: URL: https://github.com/apache/hudi/pull/4913#discussion_r1205864459 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestMarkerBasedRollbackStrategy.java: ## @@ -87,18 +88,58 @@ public void tearDown() throws

[GitHub] [hudi] xushiyan closed pull request #8799: [DNM][Test CI] Hudi 3088 default spark32 3

2023-05-25 Thread via GitHub
xushiyan closed pull request #8799: [DNM][Test CI] Hudi 3088 default spark32 3 URL: https://github.com/apache/hudi/pull/8799 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] hudi-bot commented on pull request #4913: [HUDI-1517] create marker file for every log file

2023-05-25 Thread via GitHub
hudi-bot commented on PR #4913: URL: https://github.com/apache/hudi/pull/4913#issuecomment-1563265808 ## CI report: * 4cccf801a8dfab72e1f08c4f40e29c81c6e8c66c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1729

[GitHub] [hudi] hudi-bot commented on pull request #8809: [HUDI-4944] use mor solution to uri filenotfound problem

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8809: URL: https://github.com/apache/hudi/pull/8809#issuecomment-1563261416 ## CI report: * 10519b1e65cc4a6b6e397bcf206235397b5c213a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1737

[GitHub] [hudi] hudi-bot commented on pull request #8758: [HUDI-53] Implementation of record_index - a HUDI index based on the metadata table.

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8758: URL: https://github.com/apache/hudi/pull/8758#issuecomment-1563260976 ## CI report: * c8679dfb6e1ddea34c5aa19cfe7e8f55bf78abb1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1737

[GitHub] [hudi] hudi-bot commented on pull request #4913: [HUDI-1517] create marker file for every log file

2023-05-25 Thread via GitHub
hudi-bot commented on PR #4913: URL: https://github.com/apache/hudi/pull/4913#issuecomment-1563256314 ## CI report: * 4cccf801a8dfab72e1f08c4f40e29c81c6e8c66c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1729

[GitHub] [hudi] hudi-bot commented on pull request #8809: [HUDI-4944] use mor solution to uri filenotfound problem

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8809: URL: https://github.com/apache/hudi/pull/8809#issuecomment-1563251839 ## CI report: * 10519b1e65cc4a6b6e397bcf206235397b5c213a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #8445: [HUDI-3088] Use Spark 3.2 as default Spark version

2023-05-25 Thread via GitHub
hudi-bot commented on PR #8445: URL: https://github.com/apache/hudi/pull/8445#issuecomment-1563250522 ## CI report: * 1f9f158675ba301312206710df2fad27982bc0b3 UNKNOWN * 540705cd0de0ff0f59d8246042325810c3d6b161 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] yihua commented on pull request #8669: [HUDI-5362] Rebase IncrementalRelation over HoodieBaseRelation

2023-05-25 Thread via GitHub
yihua commented on PR #8669: URL: https://github.com/apache/hudi/pull/8669#issuecomment-1563250005 Feel free to review that PR too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [hudi] yihua commented on pull request #8669: [HUDI-5362] Rebase IncrementalRelation over HoodieBaseRelation

2023-05-25 Thread via GitHub
yihua commented on PR #8669: URL: https://github.com/apache/hudi/pull/8669#issuecomment-1563249411 Hey @boneanxs no worries! I guess my PR is not fully reviewed and it's left behind. Should I apply the difference on #6045 and close this one? -- This is an automated message from the Apac

[GitHub] [hudi] guanziyue commented on a diff in pull request #4913: [HUDI-1517] create marker file for every log file

2023-05-25 Thread via GitHub
guanziyue commented on code in PR #4913: URL: https://github.com/apache/hudi/pull/4913#discussion_r1205794542 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/DefaultHoodieLogFileWriteCallBack.java: ## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundat

[GitHub] [hudi] guanziyue commented on a diff in pull request #4913: [HUDI-1517] create marker file for every log file

2023-05-25 Thread via GitHub
guanziyue commented on code in PR #4913: URL: https://github.com/apache/hudi/pull/4913#discussion_r1205793997 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestMarkerBasedRollbackStrategy.java: ## @@ -87,18 +88,58 @@ public void tearDown() throw

[GitHub] [hudi] yihua commented on a diff in pull request #8782: [HUDI-6201] use concurrent map when possible in filesystemview

2023-05-25 Thread via GitHub
yihua commented on code in PR #8782: URL: https://github.com/apache/hudi/pull/8782#discussion_r1205786475 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieTableFileSystemView.java: ## @@ -199,7 +201,7 @@ protected boolean isPendingCompactionScheduledForFileI

[GitHub] [hudi] guanziyue commented on a diff in pull request #4913: [HUDI-1517] create marker file for every log file

2023-05-25 Thread via GitHub
guanziyue commented on code in PR #4913: URL: https://github.com/apache/hudi/pull/4913#discussion_r1205732160 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkAppendHandle.java: ## @@ -60,17 +63,28 @@ public FlinkAppendHandle( Iterator> recordItr,

[GitHub] [hudi] xushiyan commented on a diff in pull request #8445: [HUDI-3088] Use Spark 3.2 as default Spark version

2023-05-25 Thread via GitHub
xushiyan commented on code in PR #8445: URL: https://github.com/apache/hudi/pull/8445#discussion_r1205766323 ## pom.xml: ## @@ -1961,27 +1965,6 @@ - -org.jacoco -jacoco-maven-plugin - -

  1   2   >