(hudi-rs) branch main updated: chore: configure codecov (#50)

2024-07-05 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/hudi-rs.git The following commit(s) were added to refs/heads/main by this push: new 5ab6361 chore: configure codecov (#50) 5ab6361

Re: [PR] chore: configure codecov target and threshold [hudi-rs]

2024-07-05 Thread via GitHub
xushiyan merged PR #50: URL: https://github.com/apache/hudi-rs/pull/50 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] chore: configure codecov target and threshold [hudi-rs]

2024-07-05 Thread via GitHub
codecov[bot] commented on PR #50: URL: https://github.com/apache/hudi-rs/pull/50#issuecomment-2211662158 ## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/50?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All

[PR] chore: configure codecov target and threshold [hudi-rs]

2024-07-05 Thread via GitHub
xushiyan opened a new pull request, #50: URL: https://github.com/apache/hudi-rs/pull/50 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[jira] [Updated] (HUDI-7963) Avoid generating RLI records when disabled w/ MDT

2024-07-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7963: - Labels: pull-request-available (was: ) > Avoid generating RLI records when disabled w/ MDT >

[PR] [HUDI-7963] Minor enhancement to RLI flow with MDT [hudi]

2024-07-05 Thread via GitHub
nsivabalan opened a new pull request, #11582: URL: https://github.com/apache/hudi/pull/11582 ### Change Logs Minor enhancement to RLI flow with MDT ### Impact Minor enhancement to RLI flow with MDT. ### Risk level (write none, low medium or high below) low

Re: [PR] [MINOR][DO NOT MERGE] Create release branch for version 1.0.0-beta2 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11558: URL: https://github.com/apache/hudi/pull/11558#issuecomment-2211649762 ## CI report: * 7e49d6fb6277247c076f73d530b66206e28f5677 Azure:

[jira] [Created] (HUDI-7963) Avoid generating RLI records when disabled w/ MDT

2024-07-05 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-7963: - Summary: Avoid generating RLI records when disabled w/ MDT Key: HUDI-7963 URL: https://issues.apache.org/jira/browse/HUDI-7963 Project: Apache Hudi

Re: [PR] [MINOR][DO NOT MERGE] Create release branch for version 1.0.0-beta2 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11558: URL: https://github.com/apache/hudi/pull/11558#issuecomment-2211647389 ## CI report: * 7e49d6fb6277247c076f73d530b66206e28f5677 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

(hudi-rs) branch main updated: feat: add config validation when creating table (#49)

2024-07-05 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/hudi-rs.git The following commit(s) were added to refs/heads/main by this push: new f1c2818 feat: add config validation when

Re: [PR] feat: add config validation when creating table [hudi-rs]

2024-07-05 Thread via GitHub
xushiyan merged PR #49: URL: https://github.com/apache/hudi-rs/pull/49 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Guard table create by validation [hudi-rs]

2024-07-05 Thread via GitHub
xushiyan closed issue #40: Guard table create by validation URL: https://github.com/apache/hudi-rs/issues/40 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] feat: add config validation when creating table [hudi-rs]

2024-07-05 Thread via GitHub
codecov[bot] commented on PR #49: URL: https://github.com/apache/hudi-rs/pull/49#issuecomment-2211640177 ## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/49?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report

[PR] feat: add config validation when creating table [hudi-rs]

2024-07-05 Thread via GitHub
xushiyan opened a new pull request, #49: URL: https://github.com/apache/hudi-rs/pull/49 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] [HUDI-7929] create k8s example for flink hudi integration [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11570: URL: https://github.com/apache/hudi/pull/11570#issuecomment-2211631487 ## CI report: * a675f8ddcf54dc0e40974d7ce387626eac74877b Azure:

Re: [PR] [HUDI-7929] create k8s example for flink hudi integration [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11570: URL: https://github.com/apache/hudi/pull/11570#issuecomment-2211614189 ## CI report: * Unknown: [CANCELED](TBD) * a675f8ddcf54dc0e40974d7ce387626eac74877b Azure:

Re: [PR] [HUDI-7929] create k8s example for flink hudi integration [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11570: URL: https://github.com/apache/hudi/pull/11570#issuecomment-2211611735 ## CI report: * Unknown: [CANCELED](TBD) * a675f8ddcf54dc0e40974d7ce387626eac74877b UNKNOWN Bot commands @hudi-bot supports the following commands:

Re: [PR] [HUDI-7929] create k8s example for flink hudi integration [hudi]

2024-07-05 Thread via GitHub
HuangZhenQiu commented on PR #11570: URL: https://github.com/apache/hudi/pull/11570#issuecomment-2211610143 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [HUDI-7961] Optimizing upsert partitioner for prepped write operations [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11581: URL: https://github.com/apache/hudi/pull/11581#issuecomment-2211558316 ## CI report: * f1a41c985871b1cca147c1f56f44ef9b1ac33dcf Azure:

Re: [PR] [HUDI-2955] Support Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11572: URL: https://github.com/apache/hudi/pull/11572#issuecomment-2211558276 ## CI report: * 83717a662e8f7defd946e519d5426f465b9bf6b1 Azure:

Re: [PR] [HUDI-7961] Optimizing upsert partitioner for prepped write operations [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11581: URL: https://github.com/apache/hudi/pull/11581#discussion_r1667217086 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/UpsertPartitioner.java: ## @@ -86,16 +87,21 @@ public class UpsertPartitioner

Re: [I] [SUPPORT] Support setting the maximum number of partitions for a table [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on issue #11566: URL: https://github.com/apache/hudi/issues/11566#issuecomment-2211545099 > file index statistics . I don't know if Hudi metrics have such planning in the future, or you could write your own logics. A valuable request, I think we can put it in the

Re: [PR] [HUDI-7955] Account for WritableTimestampObjectInspector#getPrimitive… [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11576: URL: https://github.com/apache/hudi/pull/11576#discussion_r1667216706 ## hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/Hive2Shim.java: ## @@ -42,6 +42,11 @@ public Writable getTimestampWriteable(long value, boolean

Re: [I] [SUPPORT]After restarting the program .hoodie/metadata/record_index is completely deleted [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on issue #11567: URL: https://github.com/apache/hudi/issues/11567#issuecomment-2211542159 @MrAladdin @ad1happy2go When the RLI is disabled, the dir would be purged. So we might need to find the culprit why the RLI is disabled? -- This is an automated message from the

Re: [PR] [HUDI-7929] create k8s example for flink hudi integration [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11570: URL: https://github.com/apache/hudi/pull/11570#discussion_r1667216189 ## hudi-examples/hudi-examples-k8s/src/main/java/org/apache/hudi/examples/k8s/quickstart/HudiDataStreamWriter.java: ## @@ -0,0 +1,214 @@ +/* + * Licensed to the

Re: [PR] [HUDI-7929] create k8s example for flink hudi integration [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11570: URL: https://github.com/apache/hudi/pull/11570#discussion_r1667216146 ## hudi-examples/hudi-examples-k8s/src/main/java/org/apache/hudi/examples/k8s/quickstart/HudiDataStreamWriter.java: ## @@ -0,0 +1,214 @@ +/* + * Licensed to the

Re: [PR] [HUDI-7915] Spark4 + Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11539: URL: https://github.com/apache/hudi/pull/11539#issuecomment-2211538514 ## CI report: * d1233f1a5c9c106d34babedc224ace296abe867d Azure:

[jira] [Created] (HUDI-7962) Add show create table command

2024-07-05 Thread Danny Chen (Jira)
Danny Chen created HUDI-7962: Summary: Add show create table command Key: HUDI-7962 URL: https://issues.apache.org/jira/browse/HUDI-7962 Project: Apache Hudi Issue Type: New Feature

Re: [PR] [HUDI-2955] Support Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11572: URL: https://github.com/apache/hudi/pull/11572#issuecomment-2211538605 ## CI report: * 4c35d853c40ce32f4b5a99e494e89922f32196e6 Azure:

Re: [PR] [HUDI-2955] Support Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11572: URL: https://github.com/apache/hudi/pull/11572#issuecomment-2211533698 ## CI report: * 4c35d853c40ce32f4b5a99e494e89922f32196e6 Azure:

Re: [PR] [HUDI-7915] Spark4 + Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11539: URL: https://github.com/apache/hudi/pull/11539#issuecomment-2211533604 ## CI report: * 0f34075a13d5411701af7babd6320a3619ea9981 Azure:

Re: [PR] [MINOR] Removing dead code in HiveAvroSerializer [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on PR #11577: URL: https://github.com/apache/hudi/pull/11577#issuecomment-2211530303 Thanks for the contribution, can you check the Azure CI failures? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

(hudi) branch master updated (4b52e27eb3e -> dbfe8b23c0b)

2024-07-05 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 4b52e27eb3e [HUDI-7954] Fix data skipping with secondary index when there are no log files (#11575) add

Re: [PR] [HUDI-7953] Improved the variable naming and formatting of HoodieActiveTimeline and HoodieIndex [hudi]

2024-07-05 Thread via GitHub
danny0405 merged PR #11574: URL: https://github.com/apache/hudi/pull/11574 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11545: URL: https://github.com/apache/hudi/pull/11545#discussion_r1667211813 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java: ## @@ -338,18 +339,22 @@ protected void deleteInstantFile(HoodieInstant

Re: [PR] [HUDI-7859] Rename instant files to be consistent with 0.x naming format when downgrade [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11545: URL: https://github.com/apache/hudi/pull/11545#discussion_r1667211548 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java: ## @@ -140,13 +140,19 @@ public void run(HoodieTableVersion

Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11568: URL: https://github.com/apache/hudi/pull/11568#discussion_r1667210894 ## hudi-spark-datasource/hudi-spark3.1.x/src/main/scala/org/apache/spark/sql/HoodieSpark31CatalystPlanUtils.scala: ## @@ -83,4 +82,13 @@ object

Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11568: URL: https://github.com/apache/hudi/pull/11568#discussion_r1667210845 ## hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala: ## @@ -56,15 +57,6 @@ trait

Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11568: URL: https://github.com/apache/hudi/pull/11568#discussion_r1667210673 ## hudi-spark-datasource/hudi-spark2/src/main/scala/org/apache/spark/sql/HoodieSpark2CatalystPlanUtils.scala: ## @@ -61,10 +61,10 @@ object

Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11568: URL: https://github.com/apache/hudi/pull/11568#discussion_r1667210457 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalystPlansUtils.scala: ## @@ -112,7 +113,7 @@ trait HoodieCatalystPlansUtils { *

Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11568: URL: https://github.com/apache/hudi/pull/11568#discussion_r1667210583 ## hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala: ## @@ -408,12 +408,20 @@ case class

Re: [PR] [HUDI-7957] fix data skew when writing with bulk_insert + bucket_inde… [hudi]

2024-07-05 Thread via GitHub
danny0405 commented on code in PR #11578: URL: https://github.com/apache/hudi/pull/11578#discussion_r1667210143 ## hudi-common/src/main/java/org/apache/hudi/common/util/hash/BucketIndexUtil.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] [HUDI-7961] Optimizing upsert partitioner for prepped write operations [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11581: URL: https://github.com/apache/hudi/pull/11581#issuecomment-2211502618 ## CI report: * f1a41c985871b1cca147c1f56f44ef9b1ac33dcf Azure:

Re: [PR] [HUDI-7961] Optimizing upsert partitioner for prepped write operations [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11581: URL: https://github.com/apache/hudi/pull/11581#issuecomment-2211499624 ## CI report: * f1a41c985871b1cca147c1f56f44ef9b1ac33dcf UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-2955] Support Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11572: URL: https://github.com/apache/hudi/pull/11572#issuecomment-2211499598 ## CI report: * 4c35d853c40ce32f4b5a99e494e89922f32196e6 Azure:

Re: [PR] [HUDI-7915] Spark4 + Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11539: URL: https://github.com/apache/hudi/pull/11539#issuecomment-2211499538 ## CI report: * 0f34075a13d5411701af7babd6320a3619ea9981 Azure:

Re: [PR] [HUDI-7915] Spark4 + Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11539: URL: https://github.com/apache/hudi/pull/11539#issuecomment-2211496428 ## CI report: * 57e40251eba6a0d7dc68cd10b832478f4d2decb3 Azure:

[jira] [Created] (HUDI-7961) Optimize UpsertPartitioner for prepped write operations

2024-07-05 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-7961: - Summary: Optimize UpsertPartitioner for prepped write operations Key: HUDI-7961 URL: https://issues.apache.org/jira/browse/HUDI-7961 Project: Apache Hudi

[jira] [Updated] (HUDI-7961) Optimize UpsertPartitioner for prepped write operations

2024-07-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7961: - Labels: pull-request-available (was: ) > Optimize UpsertPartitioner for prepped write operations

[PR] [HUDI-7961] Optimizing upsert partitioner for prepped write operations [hudi]

2024-07-05 Thread via GitHub
nsivabalan opened a new pull request, #11581: URL: https://github.com/apache/hudi/pull/11581 ### Change Logs Optimizing upsert partitioner for prepped write operations. also, MDT could also leverage the optimization. ### Impact Minor improvement in writes. ###

Re: [PR] [HUDI-2955] Support Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11572: URL: https://github.com/apache/hudi/pull/11572#issuecomment-2211471632 ## CI report: * 3e3ed08411c0d5ba73fdb59c06cab74ae0996acc Azure:

Re: [PR] [HUDI-2955] Support Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11572: URL: https://github.com/apache/hudi/pull/11572#issuecomment-2211467797 ## CI report: * 3e3ed08411c0d5ba73fdb59c06cab74ae0996acc Azure:

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11344: URL: https://github.com/apache/hudi/pull/11344#issuecomment-2211421877 ## CI report: * e9c58f48cb1d142f18f362632af28dcf651b51a4 Azure:

Re: [PR] [HUDI-2955] Support Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11572: URL: https://github.com/apache/hudi/pull/11572#issuecomment-2211422175 ## CI report: * 3e3ed08411c0d5ba73fdb59c06cab74ae0996acc Azure:

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11580: URL: https://github.com/apache/hudi/pull/11580#issuecomment-2211384582 ## CI report: * 3a1c57e3dc77d325881e8093a72bff4927cad160 UNKNOWN * 4c398cad0f693636a91f7797e24c5398b3122afe Azure:

Re: [PR] [HUDI-2955] Support Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11572: URL: https://github.com/apache/hudi/pull/11572#issuecomment-2211384517 ## CI report: * 3e3ed08411c0d5ba73fdb59c06cab74ae0996acc Azure:

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11344: URL: https://github.com/apache/hudi/pull/11344#issuecomment-2211384069 ## CI report: * 0397e80a3f71a4c9180a08cdd03ad16d7f313661 Azure:

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11580: URL: https://github.com/apache/hudi/pull/11580#issuecomment-2211378245 ## CI report: * 3a1c57e3dc77d325881e8093a72bff4927cad160 UNKNOWN * 4c398cad0f693636a91f7797e24c5398b3122afe UNKNOWN Bot commands @hudi-bot supports the

Re: [PR] [HUDI-2955] Support Hadoop3 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11572: URL: https://github.com/apache/hudi/pull/11572#issuecomment-2211378187 ## CI report: * 3e3ed08411c0d5ba73fdb59c06cab74ae0996acc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11344: URL: https://github.com/apache/hudi/pull/11344#issuecomment-2211377788 ## CI report: * 0397e80a3f71a4c9180a08cdd03ad16d7f313661 Azure:

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11580: URL: https://github.com/apache/hudi/pull/11580#issuecomment-2211371771 ## CI report: * 3a1c57e3dc77d325881e8093a72bff4927cad160 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-6510] [WIP] Support compilation on Java 17 [hudi]

2024-07-05 Thread via GitHub
CTTY commented on PR #11573: URL: https://github.com/apache/hudi/pull/11573#issuecomment-2211354159 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [HUDI-2955] Support Hadoop3 [hudi]

2024-07-05 Thread via GitHub
CTTY commented on PR #11572: URL: https://github.com/apache/hudi/pull/11572#issuecomment-2211353982 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-07-05 Thread via GitHub
nsivabalan commented on code in PR #11344: URL: https://github.com/apache/hudi/pull/11344#discussion_r1667092643 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java: ## @@ -908,6 +909,14 @@ public void validateInsertSchema() throws

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-07-05 Thread via GitHub
nsivabalan commented on code in PR #11344: URL: https://github.com/apache/hudi/pull/11344#discussion_r1667085731 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -924,11 +925,11 @@ private void startCommit(String

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-07-05 Thread via GitHub
nsivabalan commented on code in PR #11344: URL: https://github.com/apache/hudi/pull/11344#discussion_r1667085731 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -924,11 +925,11 @@ private void startCommit(String

[PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-07-05 Thread via GitHub
nsivabalan opened a new pull request, #11580: URL: https://github.com/apache/hudi/pull/11580 ### Change Logs When multiple writers trigger table services, there is a chance that one of them could create requested in a different ordering compared to the actual timestamp. Linked jira

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested timeli… [hudi]

2024-07-05 Thread via GitHub
nsivabalan commented on PR #11344: URL: https://github.com/apache/hudi/pull/11344#issuecomment-221132 here is the patch for 0.x branch https://github.com/apache/hudi/pull/11580 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[I] Write contribution guide and document dev setup [hudi-rs]

2024-07-05 Thread via GitHub
xushiyan opened a new issue, #44: URL: https://github.com/apache/hudi-rs/issues/44 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-7905] Use cluster action for clustering pending instants [hudi]

2024-07-05 Thread via GitHub
nsivabalan commented on code in PR #11553: URL: https://github.com/apache/hudi/pull/11553#discussion_r1667079266 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java: ## @@ -880,8 +872,7 @@ protected Map>

Re: [PR] [HUDI-7929] create k8s example for flink hudi integration [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11570: URL: https://github.com/apache/hudi/pull/11570#issuecomment-2211319889 ## CI report: * e7b4d785875ad4efd588dd24e2cefbabeb081a8b Azure:

Re: [PR] [MINOR][DO NOT MERGE] Create release branch for version 1.0.0-beta2 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11558: URL: https://github.com/apache/hudi/pull/11558#issuecomment-2211312610 ## CI report: * 7e49d6fb6277247c076f73d530b66206e28f5677 Azure:

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested timeli… [hudi]

2024-07-05 Thread via GitHub
nsivabalan commented on PR #11344: URL: https://github.com/apache/hudi/pull/11344#issuecomment-2211301073 hey @danny0405 : yes. I will raise a patch against 0.x branch. we may not need it for 1.x. Or we can debate if its required for 1.x. but for 0.x branch. we definitely need it. --

[jira] [Assigned] (HUDI-7960) Support more partitioner in Hudi Flink integration

2024-07-05 Thread Shiyan Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiyan Xu reassigned HUDI-7960: --- Assignee: Zhenqiu Huang > Support more partitioner in Hudi Flink integration >

Re: [PR] [HUDI-7929] create k8s example for flink hudi integration [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11570: URL: https://github.com/apache/hudi/pull/11570#issuecomment-2211268501 ## CI report: * dd266eb4946507ad37ccb7a5bff6071fcd31d9d9 Azure:

Re: [PR] [HUDI-7929] create k8s example for flink hudi integration [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11570: URL: https://github.com/apache/hudi/pull/11570#issuecomment-2211260146 ## CI report: * dd266eb4946507ad37ccb7a5bff6071fcd31d9d9 Azure:

Re: [PR] [MINOR][DO NOT MERGE] Create release branch for version 1.0.0-beta2 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11558: URL: https://github.com/apache/hudi/pull/11558#issuecomment-2211251875 ## CI report: * 409cea74386b555a285f72e70c7765f18407bc8a Azure:

[jira] [Created] (HUDI-7960) Support more partitioner in Hudi Flink integration

2024-07-05 Thread Zhenqiu Huang (Jira)
Zhenqiu Huang created HUDI-7960: --- Summary: Support more partitioner in Hudi Flink integration Key: HUDI-7960 URL: https://issues.apache.org/jira/browse/HUDI-7960 Project: Apache Hudi Issue

Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11579: URL: https://github.com/apache/hudi/pull/11579#issuecomment-2211188134 ## CI report: * 25c6ae81e406de37846d79655e8949e49eef6806 Azure:

Re: [PR] [MINOR][DO NOT MERGE] Create release branch for version 1.0.0-beta2 [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11558: URL: https://github.com/apache/hudi/pull/11558#issuecomment-2211188069 ## CI report: * 409cea74386b555a285f72e70c7765f18407bc8a Azure:

Re: [I] [SUPPORT] Hudi Metadata Compaction is not happeing [hudi]

2024-07-05 Thread via GitHub
xushiyan commented on issue #11535: URL: https://github.com/apache/hudi/issues/11535#issuecomment-2211178125 > run compaction of the metadata table asynchrounously no option to do that as MT compaction is managed internally > `hoodie.metadata.max.deltacommits.when_pending`

Re: [PR] [HUDI-7957] fix data skew when writing with bulk_insert + bucket_inde… [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11578: URL: https://github.com/apache/hudi/pull/11578#issuecomment-2211147856 ## CI report: * 91944ec23245ca7389fb2e36f3d96fd255a6d77a Azure:

Re: [PR] [HUDI-7921] Fixing file system view closures in MDT [hudi]

2024-07-05 Thread via GitHub
lokeshj1703 commented on code in PR #11496: URL: https://github.com/apache/hudi/pull/11496#discussion_r1666742399 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java: ## @@ -154,6 +154,9 @@ private void initIfNeeded() {

[jira] [Closed] (HUDI-7954) Fix data skipping with secondary index when there are no log files

2024-07-05 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-7954. - Resolution: Fixed > Fix data skipping with secondary index when there are no log files >

(hudi) branch master updated: [HUDI-7954] Fix data skipping with secondary index when there are no log files (#11575)

2024-07-05 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 4b52e27eb3e [HUDI-7954] Fix data skipping with

Re: [PR] [HUDI-7954] Fix data skipping with secondary index when there are no log files [hudi]

2024-07-05 Thread via GitHub
codope merged PR #11575: URL: https://github.com/apache/hudi/pull/11575 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-7954] Fix data skipping with secondary index when there are no log files [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11575: URL: https://github.com/apache/hudi/pull/11575#issuecomment-2211139566 ## CI report: * a7a939095743232ff650d6c1a6607bcf5de645b8 Azure:

Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11579: URL: https://github.com/apache/hudi/pull/11579#issuecomment-2211139635 ## CI report: * 25c6ae81e406de37846d79655e8949e49eef6806 Azure:

Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11579: URL: https://github.com/apache/hudi/pull/11579#issuecomment-2211130646 ## CI report: * 25c6ae81e406de37846d79655e8949e49eef6806 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7954] Fix data skipping with secondary index when there are no log files [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11575: URL: https://github.com/apache/hudi/pull/11575#issuecomment-2211130559 ## CI report: * a7a939095743232ff650d6c1a6607bcf5de645b8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7954] Fix data skipping with secondary index when there are no log files [hudi]

2024-07-05 Thread via GitHub
codope commented on PR #11575: URL: https://github.com/apache/hudi/pull/11575#issuecomment-2211127715 > I feel its high time we add abstractions to HoodieBackedTableMetadata. most of reading from base, logs and merging them can be abstracted out to avoid such bugs when introducing new

[jira] [Created] (HUDI-7959) Refactor HoodieBackedTableMetadata APIs

2024-07-05 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-7959: - Summary: Refactor HoodieBackedTableMetadata APIs Key: HUDI-7959 URL: https://issues.apache.org/jira/browse/HUDI-7959 Project: Apache Hudi Issue Type: Task

Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]

2024-07-05 Thread via GitHub
nsivabalan commented on code in PR #11579: URL: https://github.com/apache/hudi/pull/11579#discussion_r1666960268 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPartitionStatsIndexWithSql.scala: ## @@ -261,41 +261,75 @@ class

Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]

2024-07-05 Thread via GitHub
nsivabalan commented on code in PR #11579: URL: https://github.com/apache/hudi/pull/11579#discussion_r1666955280 ## hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java: ## @@ -332,7 +332,7 @@ public final class HoodieMetadataConfig extends

Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]

2024-07-05 Thread via GitHub
nsivabalan commented on PR #11579: URL: https://github.com/apache/hudi/pull/11579#issuecomment-2211095975 I really feel we should cut down on the no of cols we generate stats out of the box. I have encountered OSS users give col stats a try and since it takes lot of time to populate col

Re: [PR] [HUDI-7957] fix data skew when writing with bulk_insert + bucket_inde… [hudi]

2024-07-05 Thread via GitHub
hudi-bot commented on PR #11578: URL: https://github.com/apache/hudi/pull/11578#issuecomment-2211073160 ## CI report: * 91944ec23245ca7389fb2e36f3d96fd255a6d77a Azure:

Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]

2024-07-05 Thread via GitHub
codope commented on code in PR #11579: URL: https://github.com/apache/hudi/pull/11579#discussion_r1666935047 ## hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java: ## @@ -332,7 +332,7 @@ public final class HoodieMetadataConfig extends

[jira] [Updated] (HUDI-7958) Create partition stats index for all columns when no columns specified

2024-07-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7958: - Labels: pull-request-available (was: ) > Create partition stats index for all columns when no

[PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]

2024-07-05 Thread via GitHub
codope opened a new pull request, #11579: URL: https://github.com/apache/hudi/pull/11579 ### Change Logs Just like column stats index, we can create partition stats index for all column if no columns configured by the user. ### Impact Users don't necessarily have to

[jira] [Created] (HUDI-7958) Create partition stats index for all columns when no columns specified

2024-07-05 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-7958: - Summary: Create partition stats index for all columns when no columns specified Key: HUDI-7958 URL: https://issues.apache.org/jira/browse/HUDI-7958 Project: Apache Hudi

  1   2   >