[jira] [Updated] (HUDI-7513) Add jackson-module-scala to spark bundle

2024-03-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7513: - Labels: pull-request-available (was: ) > Add jackson-module-scala to spark bundle > -

[PR] [HUDI-7513] Add jackson-module-scala to spark bundle [hudi]

2024-03-17 Thread via GitHub
xicm opened a new pull request, #10877: URL: https://github.com/apache/hudi/pull/10877 ### Change Logs #10097 we relocated "com.fasterxml.jackson.module", this changes the package path of jackson-module-scala. ### Impact none ### Risk level (write none, low medium

[jira] [Updated] (HUDI-7513) Add jackson-module-scala to spark bundle

2024-03-17 Thread xi chaomin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xi chaomin updated HUDI-7513: - Description: When I do spark stream reading, get NoClassDefFoundError. {code:java} // code placeholder 24

[jira] [Updated] (HUDI-7513) Add jackson-module-scala to spark bundle

2024-03-17 Thread xi chaomin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xi chaomin updated HUDI-7513: - Affects Version/s: 0.14.1 > Add jackson-module-scala to spark bundle > ---

[jira] [Updated] (HUDI-7513) Add jackson-module-scala to spark bundle

2024-03-17 Thread xi chaomin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xi chaomin updated HUDI-7513: - Description: When I do spark stream read, get NoClassDefFoundError. {code:java} // code placeholder 24/03

[jira] [Created] (HUDI-7513) Add jackson-module-scala to spark bundle

2024-03-17 Thread xi chaomin (Jira)
xi chaomin created HUDI-7513: Summary: Add jackson-module-scala to spark bundle Key: HUDI-7513 URL: https://issues.apache.org/jira/browse/HUDI-7513 Project: Apache Hudi Issue Type: Bug

Re: [PR] [HUDI-7493] Consistent naming of Cleaner configuration parameters [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10851: URL: https://github.com/apache/hudi/pull/10851#issuecomment-2003006159 ## CI report: * 7bce9399d616a570e8a04c783b06e7e2f404dc5a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

[jira] [Updated] (HUDI-7512) Support sorting of input records in insert operation

2024-03-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7512: - Labels: pull-request-available (was: ) > Support sorting of input records in insert operation > -

[PR] [HUDI-7512] sort input records for insert operation [hudi]

2024-03-17 Thread via GitHub
bhat-vinay opened a new pull request, #10876: URL: https://github.com/apache/hudi/pull/10876 ### Change Logs Allows for sorting input records in insert operation. This is still a in-progress PR - uploading to get some test signals. Pending: Custom sort columns, more unit tests

Re: [PR] [HUDI-7493] Consistent naming of Cleaner configuration parameters [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10851: URL: https://github.com/apache/hudi/pull/10851#issuecomment-2002998943 ## CI report: * 7bce9399d616a570e8a04c783b06e7e2f404dc5a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

[jira] [Assigned] (HUDI-7512) Support sorting of input records in insert operation

2024-03-17 Thread Vinaykumar Bhat (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinaykumar Bhat reassigned HUDI-7512: - Assignee: Vinaykumar Bhat > Support sorting of input records in insert operation > --

[jira] [Created] (HUDI-7512) Support sorting of input records in insert operation

2024-03-17 Thread Vinaykumar Bhat (Jira)
Vinaykumar Bhat created HUDI-7512: - Summary: Support sorting of input records in insert operation Key: HUDI-7512 URL: https://issues.apache.org/jira/browse/HUDI-7512 Project: Apache Hudi Issu

Re: [PR] [HUDI-7492] fix the issue of incorrect keygenerator specification when creating m… [hudi]

2024-03-17 Thread via GitHub
empcl commented on code in PR #10840: URL: https://github.com/apache/hudi/pull/10840#discussion_r1527887588 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieCatalog.java: ## @@ -345,7 +345,7 @@ public void createTable(ObjectPath tablePath, C

Re: [PR] [HUDI-7493] Consistent naming of Cleaner configuration parameters [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10851: URL: https://github.com/apache/hudi/pull/10851#issuecomment-2002991337 ## CI report: * 7bce9399d616a570e8a04c783b06e7e2f404dc5a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7493] Consistent naming of Cleaner configuration parameters [hudi]

2024-03-17 Thread via GitHub
geserdugarov commented on code in PR #10851: URL: https://github.com/apache/hudi/pull/10851#discussion_r1527853866 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java: ## @@ -118,28 +120,32 @@ public class HoodieCleanConfig extends Hoodi

Re: [PR] [HUDI-7493] Consistent naming of Cleaner configuration parameters [hudi]

2024-03-17 Thread via GitHub
geserdugarov commented on code in PR #10851: URL: https://github.com/apache/hudi/pull/10851#discussion_r1527853866 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java: ## @@ -118,28 +120,32 @@ public class HoodieCleanConfig extends Hoodi

Re: [PR] [HUDI-7493] Consistent naming of Cleaner configuration parameters [hudi]

2024-03-17 Thread via GitHub
geserdugarov commented on code in PR #10851: URL: https://github.com/apache/hudi/pull/10851#discussion_r1527853866 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java: ## @@ -118,28 +120,32 @@ public class HoodieCleanConfig extends Hoodi

Re: [PR] [HUDI-7493] Consistent naming of Cleaner configuration parameters [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10851: URL: https://github.com/apache/hudi/pull/10851#issuecomment-2002946027 ## CI report: * 7bce9399d616a570e8a04c783b06e7e2f404dc5a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7492] fix the issue of incorrect keygenerator specification when creating m… [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10840: URL: https://github.com/apache/hudi/pull/10840#issuecomment-2002945942 ## CI report: * 7cbdcd0777fd6e9c482f5b5ce44ccdae972b98b6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7511] Fixing offset range calculation for kafka [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10875: URL: https://github.com/apache/hudi/pull/10875#issuecomment-2002940332 ## CI report: * fc1ac188627e04bca134e0da817d0f3777331483 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7492] fix the issue of incorrect keygenerator specification when creating m… [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10840: URL: https://github.com/apache/hudi/pull/10840#issuecomment-2002928468 ## CI report: * c6d233de457320d91579376f3d4669ee4dcf8f50 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7511] Fixing offset range calculation for kafka [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10875: URL: https://github.com/apache/hudi/pull/10875#issuecomment-2002895320 ## CI report: * fc1ac188627e04bca134e0da817d0f3777331483 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7492] fix the issue of incorrect keygenerator specification when creating m… [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10840: URL: https://github.com/apache/hudi/pull/10840#issuecomment-2002895189 ## CI report: * c6d233de457320d91579376f3d4669ee4dcf8f50 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7511] Fixing offset range calculation for kafka [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10875: URL: https://github.com/apache/hudi/pull/10875#issuecomment-2002889826 ## CI report: * fc1ac188627e04bca134e0da817d0f3777331483 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7492] fix the issue of incorrect keygenerator specification when creating m… [hudi]

2024-03-17 Thread via GitHub
empcl commented on code in PR #10840: URL: https://github.com/apache/hudi/pull/10840#discussion_r1527763448 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieCatalog.java: ## @@ -345,7 +345,7 @@ public void createTable(ObjectPath tablePath, C

Re: [PR] [HUDI-7510] Loosen the compaction scheduling and rollback check for MDT [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10874: URL: https://github.com/apache/hudi/pull/10874#issuecomment-2002884545 ## CI report: * 46e36c45556766aea812b45b8f4fa7aec27e9bc0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7492] fix the issue of incorrect keygenerator specification when creating m… [hudi]

2024-03-17 Thread via GitHub
empcl commented on code in PR #10840: URL: https://github.com/apache/hudi/pull/10840#discussion_r1527763448 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieCatalog.java: ## @@ -345,7 +345,7 @@ public void createTable(ObjectPath tablePath, C

[jira] [Updated] (HUDI-7511) Offset range calculation in kafka should return all topic partitions

2024-03-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7511: - Labels: pull-request-available (was: ) > Offset range calculation in kafka should return all topi

[PR] [HUDI-7511] Fixing offset range calculation for kafka [hudi]

2024-03-17 Thread via GitHub
nsivabalan opened a new pull request, #10875: URL: https://github.com/apache/hudi/pull/10875 ### Change Logs after https://github.com/apache/hudi/pull/10869 got landed, we are not returning every topic partition in final ranges. But for checkpointing purpose, we need to have every ka

[jira] [Created] (HUDI-7511) Offset range calculation in kafka should return all topic partitions

2024-03-17 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-7511: - Summary: Offset range calculation in kafka should return all topic partitions Key: HUDI-7511 URL: https://issues.apache.org/jira/browse/HUDI-7511 Project:

Re: [PR] [HUDI-7510] Loosen the compaction scheduling and rollback check for MDT [hudi]

2024-03-17 Thread via GitHub
danny0405 commented on code in PR #10874: URL: https://github.com/apache/hudi/pull/10874#discussion_r1527740554 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -1407,43 +1382,7 @@ protected void cleanIfNecessar

Re: [PR] [HUDI-7510] Loosen the compaction scheduling and rollback check for MDT [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10874: URL: https://github.com/apache/hudi/pull/10874#issuecomment-2002824610 ## CI report: * 46e36c45556766aea812b45b8f4fa7aec27e9bc0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7436] Fix the conditions for determining whether the records need to be rewritten [hudi]

2024-03-17 Thread via GitHub
xiarixiaoyao commented on code in PR #10727: URL: https://github.com/apache/hudi/pull/10727#discussion_r1527736064 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/HoodieMergeHelper.java: ## @@ -202,7 +202,9 @@ private Option> composeSchemaEvo

Re: [PR] [HUDI-7510] Loosen the compaction scheduling and rollback check for MDT [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10874: URL: https://github.com/apache/hudi/pull/10874#issuecomment-2002805421 ## CI report: * 46e36c45556766aea812b45b8f4fa7aec27e9bc0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [I] [SUPPORT] Insert overwrite with replacement instant cannot execute archive [hudi]

2024-03-17 Thread via GitHub
xuzifu666 commented on issue #10873: URL: https://github.com/apache/hudi/issues/10873#issuecomment-2002776170 > @xuzifu666 When I tried the below code, it was properly archiving. Can you check below or provide your table/writer configurations. > > ``` > fake = Faker() > data = [

[jira] [Updated] (HUDI-7510) Loosen the compaction scheduling and rollback check for MDT

2024-03-17 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7510: - Labels: pull-request-available (was: ) > Loosen the compaction scheduling and rollback check for

[PR] [HUDI-7510] Loosen the compaction scheduling and rollback check for MDT [hudi]

2024-03-17 Thread via GitHub
danny0405 opened a new pull request, #10874: URL: https://github.com/apache/hudi/pull/10874 ### Change Logs This is a follow-up for #9871 . We can eliminate the restrictions for 2 reasons: 1. we now always generate fresh new compaction instant timestamp instead of concatenatin

[jira] [Created] (HUDI-7510) Loosen the compaction scheduling and rollback check for MDT

2024-03-17 Thread Danny Chen (Jira)
Danny Chen created HUDI-7510: Summary: Loosen the compaction scheduling and rollback check for MDT Key: HUDI-7510 URL: https://issues.apache.org/jira/browse/HUDI-7510 Project: Apache Hudi Issue

[jira] [Closed] (HUDI-7421) Build HoodieDeltaWriteStat using CommitMetadataUtils#getHoodieDeltaWriteStatFromPreviousStat

2024-03-17 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-7421. Resolution: Fixed Fixed via master branch: 2dcdd311245b38c0a9884338d6333212a730d310 > Build HoodieDeltaWrit

[jira] [Updated] (HUDI-7421) Build HoodieDeltaWriteStat using CommitMetadataUtils#getHoodieDeltaWriteStatFromPreviousStat

2024-03-17 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-7421: - Fix Version/s: 1.0.0 > Build HoodieDeltaWriteStat using > CommitMetadataUtils#getHoodieDeltaWriteStatFrom

Re: [PR] [HUDI-7421] Build HoodieDeltaWriteStat using HoodieDeltaWriteStat#copy [hudi]

2024-03-17 Thread via GitHub
danny0405 merged PR #10870: URL: https://github.com/apache/hudi/pull/10870 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apac

(hudi) branch master updated: [HUDI-7421] Build HoodieDeltaWriteStat using HoodieDeltaWriteStat#copy (#10870)

2024-03-17 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 2dcdd311245 [HUDI-7421] Build HoodieDeltaWriteS

Re: [I] [SUPPORT] Insert overwrite with replacement instant cannot execute archive [hudi]

2024-03-17 Thread via GitHub
ad1happy2go commented on issue #10873: URL: https://github.com/apache/hudi/issues/10873#issuecomment-2002570828 @xuzifu666 When I tried the below code, it was properly archiving. Can you check below or provide your table/writer configurations. ``` fake = Faker() data = [{"transa

Re: [PR] [HUDI-7421] Build HoodieDeltaWriteStat using HoodieDeltaWriteStat#copy [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10870: URL: https://github.com/apache/hudi/pull/10870#issuecomment-2002498312 ## CI report: * 3cd01a86a1be7ca6e0741268e668fcce08584d7a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7421] Build HoodieDeltaWriteStat using HoodieDeltaWriteStat#copy [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10870: URL: https://github.com/apache/hudi/pull/10870#issuecomment-2002458831 ## CI report: * 89d952bd9a1db73835787f1f0570fc922a078456 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7421] Build HoodieDeltaWriteStat using HoodieDeltaWriteStat#copy [hudi]

2024-03-17 Thread via GitHub
hudi-bot commented on PR #10870: URL: https://github.com/apache/hudi/pull/10870#issuecomment-2002456409 ## CI report: * 89d952bd9a1db73835787f1f0570fc922a078456 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22

Re: [PR] [HUDI-7421] Build HoodieDeltaWriteStat using HoodieDeltaWriteStat#copy [hudi]

2024-03-17 Thread via GitHub
wombatu-kun commented on code in PR #10870: URL: https://github.com/apache/hudi/pull/10870#discussion_r1527501760 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java: ## @@ -326,12 +327,7 @@ private MetadataValues populateMetadataFields(Ho

Re: [PR] [HUDI-7421] Build HoodieDeltaWriteStat using HoodieDeltaWriteStat#copy [hudi]

2024-03-17 Thread via GitHub
wombatu-kun commented on code in PR #10870: URL: https://github.com/apache/hudi/pull/10870#discussion_r1527500936 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java: ## @@ -326,12 +327,7 @@ private MetadataValues populateMetadataFields(Ho

[jira] [Created] (HUDI-7509) CDC functionality not available: CDC write and CDC query

2024-03-17 Thread chenxusheng (Jira)
chenxusheng created HUDI-7509: - Summary: CDC functionality not available: CDC write and CDC query Key: HUDI-7509 URL: https://issues.apache.org/jira/browse/HUDI-7509 Project: Apache Hudi Issue Ty