Re: [PR] [HUDI-6960] Support read partition values from path when schema evolution enabled [hudi]

2023-10-21 Thread via GitHub
wecharyu commented on code in PR #9889: URL: https://github.com/apache/hudi/pull/9889#discussion_r1367842247 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala: ## @@ -149,27 +152,10 @@ case class BaseFileOnlyRelation(override

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-21 Thread via GitHub
beyond1920 commented on PR #9896: URL: https://github.com/apache/hudi/pull/9896#issuecomment-1773971420 @danny0405 Thanks for review. I would update the PR later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9876: URL: https://github.com/apache/hudi/pull/9876#issuecomment-1773965020 ## CI report: * b8bc65dc87cfd1305634bf16f96a97944ce85816 Azure:

Re: [PR] [HUDI-6801] Implement merging partial updates from log files for MOR tables [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9883: URL: https://github.com/apache/hudi/pull/9883#issuecomment-1773965031 ## CI report: * d1a976ddc81d1aa79df06fcbf72acd59c2c2b518 Azure:

Re: [PR] [HUDI-6929] Lazy loading dynamically for CompletionTimeQueryView [hudi]

2023-10-21 Thread via GitHub
danny0405 commented on code in PR #9898: URL: https://github.com/apache/hudi/pull/9898#discussion_r1367819628 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java: ## @@ -157,17 +158,23 @@ public Option getCompletionTime(String

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9892: URL: https://github.com/apache/hudi/pull/9892#issuecomment-1773957168 ## CI report: * 8a3d6d6da7b0df80240d8621c4de9469a7efb1cf UNKNOWN * 355a341791a82a572e0595b44ed51372a7ee2c6e Azure:

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9876: URL: https://github.com/apache/hudi/pull/9876#issuecomment-1773948275 ## CI report: * 794904512405851fa42c10927c315ca55d82fbdc Azure:

Re: [PR] [HUDI-6801] Implement merging partial updates from log files for MOR tables [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9883: URL: https://github.com/apache/hudi/pull/9883#issuecomment-1773948280 ## CI report: * d1a976ddc81d1aa79df06fcbf72acd59c2c2b518 Azure:

Re: [PR] [HUDI-6801] Implement merging partial updates from log files for MOR tables [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9883: URL: https://github.com/apache/hudi/pull/9883#issuecomment-1773947290 ## CI report: * d1a976ddc81d1aa79df06fcbf72acd59c2c2b518 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9876: URL: https://github.com/apache/hudi/pull/9876#issuecomment-1773947285 ## CI report: * 794904512405851fa42c10927c315ca55d82fbdc Azure:

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9892: URL: https://github.com/apache/hudi/pull/9892#issuecomment-1773940182 ## CI report: * 8a3d6d6da7b0df80240d8621c4de9469a7efb1cf UNKNOWN * 088db0157670aba4efb06c273388ef99656a8ce0 Azure:

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-21 Thread via GitHub
yihua commented on code in PR #9876: URL: https://github.com/apache/hudi/pull/9876#discussion_r1367807590 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala: ## @@ -449,21 +466,58 @@ case class

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9892: URL: https://github.com/apache/hudi/pull/9892#issuecomment-1773938907 ## CI report: * 8a3d6d6da7b0df80240d8621c4de9469a7efb1cf UNKNOWN * 088db0157670aba4efb06c273388ef99656a8ce0 Azure:

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-21 Thread via GitHub
yihua commented on code in PR #9876: URL: https://github.com/apache/hudi/pull/9876#discussion_r1367807326 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/payload/ExpressionPayload.scala: ## @@ -411,10 +414,14 @@ object ExpressionPayload {

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-21 Thread via GitHub
yihua commented on code in PR #9876: URL: https://github.com/apache/hudi/pull/9876#discussion_r1367806982 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java: ## @@ -652,6 +660,16 @@ private static Map getUpdatedHeader(Map

Re: [PR] [HUDI-6800] Support writing partial updates to the data blocks in MOR tables [hudi]

2023-10-21 Thread via GitHub
yihua commented on code in PR #9876: URL: https://github.com/apache/hudi/pull/9876#discussion_r1367806866 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java: ## @@ -755,6 +756,24 @@ public class HoodieWriteConfig extends HoodieConfig {

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9892: URL: https://github.com/apache/hudi/pull/9892#issuecomment-1773937680 ## CI report: * 8a3d6d6da7b0df80240d8621c4de9469a7efb1cf UNKNOWN * 088db0157670aba4efb06c273388ef99656a8ce0 Azure:

[jira] [Updated] (HUDI-6966) Update docs on custom payload implementation

2023-10-21 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6966: Description: After HUDI-6961, we change the record payload constructor to include properties.  While the

[jira] [Updated] (HUDI-6966) Update docs on custom payload implementation

2023-10-21 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6966: Description: After HUDI-6961, we change the record payload constructor to include properties.  While the

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9892: URL: https://github.com/apache/hudi/pull/9892#issuecomment-1773930623 ## CI report: * 8a3d6d6da7b0df80240d8621c4de9469a7efb1cf UNKNOWN * e73788c46440921d480f8575bbd342a9adbf8629 Azure:

[jira] [Updated] (HUDI-6966) Update docs on custom payload implementation

2023-10-21 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6966: Fix Version/s: 0.14.1 > Update docs on custom payload implementation >

[jira] [Created] (HUDI-6966) Update docs on custom payload implementation

2023-10-21 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-6966: --- Summary: Update docs on custom payload implementation Key: HUDI-6966 URL: https://issues.apache.org/jira/browse/HUDI-6966 Project: Apache Hudi Issue Type: Improvement

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9892: URL: https://github.com/apache/hudi/pull/9892#issuecomment-1773927778 ## CI report: * 8a3d6d6da7b0df80240d8621c4de9469a7efb1cf UNKNOWN * e73788c46440921d480f8575bbd342a9adbf8629 Azure:

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9892: URL: https://github.com/apache/hudi/pull/9892#issuecomment-1773917858 ## CI report: * 8a3d6d6da7b0df80240d8621c4de9469a7efb1cf UNKNOWN * c580c03a82bb25a639e0659b779cdf10acfaf33e Azure:

Re: [PR] [HUDI-6929] Lazy loading dynamically for CompletionTimeQueryView [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9898: URL: https://github.com/apache/hudi/pull/9898#issuecomment-1773872084 ## CI report: * dcbc75c19c76cd46650052439721592167c11dec Azure:

Re: [I] [SUPPORT] PySpark reading hudi partition column for hudi table incorrectly [hudi]

2023-10-21 Thread via GitHub
bradleybonitatibus commented on issue #9890: URL: https://github.com/apache/hudi/issues/9890#issuecomment-1773841549 Thanks for the reply, @ad1happy2go! I guess for additional context, when we read this Hudi table from Redshift via external table, the `inserted_at` column reads as

Re: [PR] [HUDI-6929] Lazy loading dynamically for CompletionTimeQueryView [hudi]

2023-10-21 Thread via GitHub
codope commented on code in PR #9898: URL: https://github.com/apache/hudi/pull/9898#discussion_r1367734502 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java: ## @@ -157,17 +158,23 @@ public Option getCompletionTime(String startTime)

Re: [PR] [HUDI-6929] Lazy loading dynamically for CompletionTimeQueryView [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9898: URL: https://github.com/apache/hudi/pull/9898#issuecomment-1773797478 ## CI report: * dcbc75c19c76cd46650052439721592167c11dec Azure:

Re: [PR] [HUDI-6929] Lazy loading dynamically for CompletionTimeQueryView [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9898: URL: https://github.com/apache/hudi/pull/9898#issuecomment-1773795534 ## CI report: * dcbc75c19c76cd46650052439721592167c11dec UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9896: URL: https://github.com/apache/hudi/pull/9896#issuecomment-1773793723 ## CI report: * 6476982b71f8924be4fce2c68e67f754456b1d5d Azure:

[jira] [Updated] (HUDI-6929) Lazy loading dynamically for CompletionTimeQueryView

2023-10-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6929: - Labels: pull-request-available (was: ) > Lazy loading dynamically for CompletionTimeQueryView >

[PR] [HUDI-6929] Lazy loading dynamically for CompletionTimeQueryView [hudi]

2023-10-21 Thread via GitHub
danny0405 opened a new pull request, #9898: URL: https://github.com/apache/hudi/pull/9898 ### Change Logs Make the lazy loading more smart and efficient. ### Impact none ### Risk level (write none, low medium or high below) none ### Documentation

[jira] [Assigned] (HUDI-6929) Lazy loading dynamically for CompletionTimeQueryView

2023-10-21 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen reassigned HUDI-6929: Assignee: Danny Chen > Lazy loading dynamically for CompletionTimeQueryView >

[jira] [Updated] (HUDI-6929) Lazy loading dynamically for CompletionTimeQueryView

2023-10-21 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6929: - Fix Version/s: 1.0.0 (was: 1.1.0) > Lazy loading dynamically for

[jira] [Updated] (HUDI-6929) Lazy loading dynamically for CompletionTimeQueryView

2023-10-21 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6929: - Summary: Lazy loading dynamically for CompletionTimeQueryView (was: Make the default eager loading start

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9896: URL: https://github.com/apache/hudi/pull/9896#issuecomment-1773749041 ## CI report: * 72489cb31221dfb8152f683cd9140b401134c9ee Azure:

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-21 Thread via GitHub
hudi-bot commented on PR #9896: URL: https://github.com/apache/hudi/pull/9896#issuecomment-1773747292 ## CI report: * 72489cb31221dfb8152f683cd9140b401134c9ee Azure:

Re: [PR] [HUDI-6960] Support read partition values from path when schema evolution enabled [hudi]

2023-10-21 Thread via GitHub
danny0405 commented on code in PR #9889: URL: https://github.com/apache/hudi/pull/9889#discussion_r1367706072 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala: ## @@ -149,27 +152,10 @@ case class BaseFileOnlyRelation(override

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-21 Thread via GitHub
danny0405 commented on code in PR #9896: URL: https://github.com/apache/hudi/pull/9896#discussion_r1367703643 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java: ## @@ -2616,6 +2617,19 @@ public Integer getWritesFileIdEncoding() {

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-21 Thread via GitHub
danny0405 commented on code in PR #9896: URL: https://github.com/apache/hudi/pull/9896#discussion_r1367703369 ## hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/BulkInsertFunctionWrapper.java: ## @@ -0,0 +1,232 @@ +/* + * Licensed to the Apache

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-21 Thread via GitHub
danny0405 commented on code in PR #9896: URL: https://github.com/apache/hudi/pull/9896#discussion_r1367703360 ## hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestData.java: ## @@ -553,7 +554,12 @@ public static void writeDataAsBatch( * Initializes a

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-21 Thread via GitHub
danny0405 commented on code in PR #9896: URL: https://github.com/apache/hudi/pull/9896#discussion_r1367703247 ## hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/TestWriteBase.java: ## @@ -305,6 +305,31 @@ public TestHarness checkpointComplete(long

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-21 Thread via GitHub
danny0405 commented on code in PR #9896: URL: https://github.com/apache/hudi/pull/9896#discussion_r1367703216 ## hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/TestWriteBase.java: ## @@ -305,6 +305,31 @@ public TestHarness checkpointComplete(long

Re: [PR] [HUDI-6962] Fix the conflicts resolution for bulk insert under NB-CC [hudi]

2023-10-21 Thread via GitHub
beyond1920 commented on code in PR #9896: URL: https://github.com/apache/hudi/pull/9896#discussion_r1367698810 ## hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestData.java: ## @@ -553,7 +554,12 @@ public static void writeDataAsBatch( * Initializes a