Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
danny0405 commented on code in PR #10943: URL: https://github.com/apache/hudi/pull/10943#discussion_r1545988384 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -209,10 +210,18 @@ public static byte[] avroToJson(GenericRecord record, boolean

Re: [PR] [HUDI-7552] Remove the suffix for MDT table service instants [hudi]

2024-03-31 Thread via GitHub
hudi-bot commented on PR #10945: URL: https://github.com/apache/hudi/pull/10945#issuecomment-2029207026 ## CI report: * 1fdb25272d5d41970393eb9bc7632a697ca879af Azure:

[jira] [Closed] (HUDI-6538) Refactor methods in TimelineDiffHelper class

2024-03-31 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6538. Resolution: Fixed Fixed via master branch: 44ab6f32bffbab8cd250bd0430d9591360f118e7 > Refactor methods in

[jira] [Updated] (HUDI-6538) Refactor methods in TimelineDiffHelper class

2024-03-31 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6538: - Fix Version/s: 1.0.0 > Refactor methods in TimelineDiffHelper class >

(hudi) branch master updated: [HUDI-6538] Refactor methods in TimelineDiffHelper class (#10938)

2024-03-31 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 44ab6f32bff [HUDI-6538] Refactor methods in

Re: [PR] [HUDI-6538] Refactor methods in TimelineDiffHelper class [hudi]

2024-03-31 Thread via GitHub
danny0405 merged PR #10938: URL: https://github.com/apache/hudi/pull/10938 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Data lose after writing [hudi]

2024-03-31 Thread via GitHub
ad1happy2go commented on issue #10935: URL: https://github.com/apache/hudi/issues/10935#issuecomment-2029175623 @wangzhongz Hudi version you are using is too old. Is it possible for you to upgrade? -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [HUDI-7552] Remove the suffix for MDT table service instants [hudi]

2024-03-31 Thread via GitHub
hudi-bot commented on PR #10945: URL: https://github.com/apache/hudi/pull/10945#issuecomment-2029165793 ## CI report: * 1fdb25272d5d41970393eb9bc7632a697ca879af Azure:

Re: [PR] [HUDI-7552] Remove the suffix for MDT table service instants [hudi]

2024-03-31 Thread via GitHub
hudi-bot commented on PR #10945: URL: https://github.com/apache/hudi/pull/10945#issuecomment-2029160232 ## CI report: * 1fdb25272d5d41970393eb9bc7632a697ca879af UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

[jira] [Closed] (HUDI-7510) Loosen the compaction scheduling and rollback check for MDT

2024-03-31 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-7510. Resolution: Fixed Fixed via master branch: 9b094e628d6e4b1157cdee6e5ae951a99d32921a > Loosen the

(hudi) branch master updated (26c00a3adef -> 9b094e628d6)

2024-03-31 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 26c00a3adef [HUDI-7187] Fix integ test props to honor new streamer properties (#10866) add 9b094e628d6

Re: [PR] [HUDI-7510] Loosen the compaction scheduling and rollback check for MDT [hudi]

2024-03-31 Thread via GitHub
danny0405 merged PR #10874: URL: https://github.com/apache/hudi/pull/10874 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
the-other-tim-brown commented on code in PR #10943: URL: https://github.com/apache/hudi/pull/10943#discussion_r1545960527 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -209,10 +210,18 @@ public static byte[] avroToJson(GenericRecord record,

Re: [PR] [HUDI-6538] Refactor methods in TimelineDiffHelper class [hudi]

2024-03-31 Thread via GitHub
wombatu-kun commented on PR #10938: URL: https://github.com/apache/hudi/pull/10938#issuecomment-2029157601 @nsivabalan this refactoring is made addressing the code you proposed in comment to other PR. Could you please review it? -- This is an automated message from the Apache Git

Re: [PR] [HUDI-7526] Fix constructors for bulkinsert sort partitioners to ensure we could use it as user defined partitioners [hudi]

2024-03-31 Thread via GitHub
wombatu-kun commented on code in PR #10942: URL: https://github.com/apache/hudi/pull/10942#discussion_r1545958016 ## hudi-client/hudi-java-client/src/main/java/org/apache/hudi/execution/bulkinsert/JavaGlobalSortPartitioner.java: ## @@ -31,12 +32,21 @@ * * @param

Re: [PR] [MINOR] Changing the Properties to Load From Both Default Path and Enviorment [hudi]

2024-03-31 Thread via GitHub
Amar1404 commented on PR #10835: URL: https://github.com/apache/hudi/pull/10835#issuecomment-2029144280 @CTTY : So here In EMR the default conf is getting applied, but as per the document of hudi if we specify the ENV HUDI_DEFAULT_CONF is not getting applied due to the bug in code, which i

[jira] [Created] (HUDI-7557) NoSuchElementException when savepoint has been removed or archived

2024-03-31 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-7557: - Summary: NoSuchElementException when savepoint has been removed or archived Key: HUDI-7557 URL: https://issues.apache.org/jira/browse/HUDI-7557 Project: Apache Hudi

Re: [PR] [HUDI-7526] Fix constructors for bulkinsert sort partitioners to ensure we could use it as user defined partitioners [hudi]

2024-03-31 Thread via GitHub
danny0405 commented on code in PR #10942: URL: https://github.com/apache/hudi/pull/10942#discussion_r1545945929 ## hudi-client/hudi-java-client/src/main/java/org/apache/hudi/execution/bulkinsert/JavaGlobalSortPartitioner.java: ## @@ -31,12 +32,21 @@ * * @param

Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
danny0405 commented on code in PR #10943: URL: https://github.com/apache/hudi/pull/10943#discussion_r1545945200 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -209,10 +210,18 @@ public static byte[] avroToJson(GenericRecord record, boolean

[jira] [Updated] (HUDI-7552) Remove the suffix for MDT table service instants

2024-03-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7552: - Labels: pull-request-available (was: ) > Remove the suffix for MDT table service instants >

[PR] [HUDI-7552] Remove the suffix for MDT table service instants [hudi]

2024-03-31 Thread via GitHub
danny0405 opened a new pull request, #10945: URL: https://github.com/apache/hudi/pull/10945 ### Change Logs Remove the suffix of MDT table operation instants (the async index operation is kept because there is still some validation on it, the suffix is used for efficient filtering).

Re: [PR] [HUDI-7526] Fix constructors for bulkinsert sort partitioners to ensure we could use it as user defined partitioners [hudi]

2024-03-31 Thread via GitHub
wombatu-kun commented on code in PR #10942: URL: https://github.com/apache/hudi/pull/10942#discussion_r1545900885 ## hudi-client/hudi-java-client/src/main/java/org/apache/hudi/execution/bulkinsert/JavaGlobalSortPartitioner.java: ## @@ -31,12 +32,21 @@ * * @param

Re: [PR] [HUDI-7526] Fix constructors for bulkinsert sort partitioners to ensure we could use it as user defined partitioners [hudi]

2024-03-31 Thread via GitHub
wombatu-kun commented on code in PR #10942: URL: https://github.com/apache/hudi/pull/10942#discussion_r1545900885 ## hudi-client/hudi-java-client/src/main/java/org/apache/hudi/execution/bulkinsert/JavaGlobalSortPartitioner.java: ## @@ -31,12 +32,21 @@ * * @param

Re: [PR] [HUDI-7526] Fix constructors for bulkinsert sort partitioners to ensure we could use it as user defined partitioners [hudi]

2024-03-31 Thread via GitHub
wombatu-kun commented on code in PR #10942: URL: https://github.com/apache/hudi/pull/10942#discussion_r1545900885 ## hudi-client/hudi-java-client/src/main/java/org/apache/hudi/execution/bulkinsert/JavaGlobalSortPartitioner.java: ## @@ -31,12 +32,21 @@ * * @param

Re: [PR] [HUDI-7522] Support find out the conflict instants in bucket partition when bucket id multiple [hudi]

2024-03-31 Thread via GitHub
xuzifu666 commented on code in PR #10898: URL: https://github.com/apache/hudi/pull/10898#discussion_r1545892451 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bucket/HoodieSimpleBucketIndex.java: ## @@ -61,14 +75,63 @@ public Map

Re: [PR] [HUDI-7522] Support find out the conflict instants in bucket partition when bucket id multiple [hudi]

2024-03-31 Thread via GitHub
xuzifu666 commented on code in PR #10898: URL: https://github.com/apache/hudi/pull/10898#discussion_r1545892451 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bucket/HoodieSimpleBucketIndex.java: ## @@ -61,14 +75,63 @@ public Map

Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
hudi-bot commented on PR #10943: URL: https://github.com/apache/hudi/pull/10943#issuecomment-2029024584 ## CI report: * 70a35f705b74db87648f3f6a7e504614db6416aa Azure:

[I] [SUGGEST] Can the community version be updated regularly and faster? The roadmap should also be updated regularly and synchronized. [hudi]

2024-03-31 Thread via GitHub
zyclove opened a new issue, #10944: URL: https://github.com/apache/hudi/issues/10944 The version update is also too slow. It has not been updated for a long time. Many problems have to be solved in time and support is not available.

(hudi) branch asf-site updated: rs - cow snap, mor ro; starrocks - cow snap, mor rt, ro (#10940)

2024-03-31 Thread bhavanisudha
This is an automated email from the ASF dual-hosted git repository. bhavanisudha pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new e8d498d2199 rs - cow snap, mor ro;

Re: [PR] [MINOR] [DOCS] changes to redshift & starrocks compat matrix [hudi]

2024-03-31 Thread via GitHub
bhasudha merged PR #10940: URL: https://github.com/apache/hudi/pull/10940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
the-other-tim-brown commented on code in PR #10943: URL: https://github.com/apache/hudi/pull/10943#discussion_r1545880874 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -209,10 +210,18 @@ public static byte[] avroToJson(GenericRecord record,

Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
the-other-tim-brown commented on code in PR #10943: URL: https://github.com/apache/hudi/pull/10943#discussion_r1545880266 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -209,10 +210,18 @@ public static byte[] avroToJson(GenericRecord record,

Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
danny0405 commented on code in PR #10943: URL: https://github.com/apache/hudi/pull/10943#discussion_r1545879905 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -209,10 +210,18 @@ public static byte[] avroToJson(GenericRecord record, boolean

Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
danny0405 commented on code in PR #10943: URL: https://github.com/apache/hudi/pull/10943#discussion_r1545879712 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -209,10 +210,18 @@ public static byte[] avroToJson(GenericRecord record, boolean

Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
hudi-bot commented on PR #10943: URL: https://github.com/apache/hudi/pull/10943#issuecomment-2028989292 ## CI report: * 70a35f705b74db87648f3f6a7e504614db6416aa Azure:

Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
danny0405 commented on code in PR #10943: URL: https://github.com/apache/hudi/pull/10943#discussion_r1545879498 ## hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java: ## @@ -209,10 +210,18 @@ public static byte[] avroToJson(GenericRecord record, boolean

Re: [PR] [HUDI-7526] Fix constructors for bulkinsert sort partitioners to ensure we could use it as user defined partitioners [hudi]

2024-03-31 Thread via GitHub
danny0405 commented on code in PR #10942: URL: https://github.com/apache/hudi/pull/10942#discussion_r1545879218 ## hudi-client/hudi-java-client/src/main/java/org/apache/hudi/execution/bulkinsert/JavaGlobalSortPartitioner.java: ## @@ -31,12 +32,21 @@ * * @param

Re: [PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
hudi-bot commented on PR #10943: URL: https://github.com/apache/hudi/pull/10943#issuecomment-2028985048 ## CI report: * 70a35f705b74db87648f3f6a7e504614db6416aa UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

[PR] [MINOR] Handle cases of malformed records when converting to json [hudi]

2024-03-31 Thread via GitHub
the-other-tim-brown opened a new pull request, #10943: URL: https://github.com/apache/hudi/pull/10943 ### Change Logs Handles cases of missing required fields and bad input values when converting to JSON. This conversion is used in combination with the Error Table so you cannot

Re: [PR] [HUDI-7526] Fix constructors for bulkinsert sort partitioners to ensure we could use it as user defined partitioners [hudi]

2024-03-31 Thread via GitHub
hudi-bot commented on PR #10942: URL: https://github.com/apache/hudi/pull/10942#issuecomment-2028820679 ## CI report: * ea11f68c1778f9ec23eab6a887076e51f60caa0b Azure:

Re: [PR] [HUDI-7526] Fix constructors for bulkinsert sort partitioners to ensure we could use it as user defined partitioners [hudi]

2024-03-31 Thread via GitHub
hudi-bot commented on PR #10942: URL: https://github.com/apache/hudi/pull/10942#issuecomment-2028787238 ## CI report: * ea11f68c1778f9ec23eab6a887076e51f60caa0b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

[jira] [Updated] (HUDI-7526) Fix constructors for all bulk insert sort partitioners to ensure we could use it as user defined partitioners

2024-03-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7526: - Labels: pull-request-available (was: ) > Fix constructors for all bulk insert sort partitioners

[PR] [HUDI-7526] Fix constructors for bulkinsert sort partitioners to ensure we could use it as user defined partitioners [hudi]

2024-03-31 Thread via GitHub
wombatu-kun opened a new pull request, #10942: URL: https://github.com/apache/hudi/pull/10942 ### Change Logs Our constructor for user defined sort partitioner takes in write config, while some of the partitioners used in out of the box sort mode, does not account for it. Lets fix

[jira] [Assigned] (HUDI-7526) Fix constructors for all bulk insert sort partitioners to ensure we could use it as user defined partitioners

2024-03-31 Thread Vova Kolmakov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vova Kolmakov reassigned HUDI-7526: --- Assignee: Vova Kolmakov > Fix constructors for all bulk insert sort partitioners to ensure

[jira] [Updated] (HUDI-7526) Fix constructors for all bulk insert sort partitioners to ensure we could use it as user defined partitioners

2024-03-31 Thread Vova Kolmakov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vova Kolmakov updated HUDI-7526: Status: In Progress (was: Open) > Fix constructors for all bulk insert sort partitioners to ensure

Re: [PR] [HUDI-3397] Guard repeated rdd triggers [hudi]

2024-03-31 Thread via GitHub
bvaradar commented on PR #6878: URL: https://github.com/apache/hudi/pull/6878#issuecomment-2028572932 @nsivabalan : Is this PR still relevant or can be closed ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL