Re: [I] [SUPPORT] spark-sql write the table fail witch was created by flink-sql with HoodieHiveCatalog and the error is:Config conflict(key current value existing value): PreCombineKey: ts [hudi]

2024-07-25 Thread via GitHub
danny0405 commented on issue #11687: URL: https://github.com/apache/hudi/issues/11687#issuecomment-2251938209 Probably because you are using a legacy version of Hudi, the option key for Flink is `precombine.field` which is incompatible with Spark, we have added the compatibility code for

Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]

2024-07-25 Thread via GitHub
soumilshah1995 commented on issue #10644: URL: https://github.com/apache/hudi/issues/10644#issuecomment-2251899581 Yes you are right I will compile master branch and keep you posted Thanking You, Soumil Nitin Shah On Thu, Jul 25, 2024 at 11:29 PM Vova Kolmakov

Re: [I] [SUPPORT] spark-sql write the table fail witch was created by flink-sql with HoodieHiveCatalog and the error is:Config conflict(key current value existing value): PreCombineKey: ts [hudi]

2024-07-25 Thread via GitHub
wardlican commented on issue #11687: URL: https://github.com/apache/hudi/issues/11687#issuecomment-2251854701 > hoodie.datasource.write.precombine.field Thank you very much for your help. I tested that after adding hoodie.datasource.write.precombine.field=ts when creating the table

Re: [PR] ci: fix running tests on relevant paths [hudi-rs]

2024-07-25 Thread via GitHub
xushiyan commented on code in PR #93: URL: https://github.com/apache/hudi-rs/pull/93#discussion_r1692418271 ## .github/workflows/ci.yml: ## @@ -22,9 +22,17 @@ on: branches: - main - release/** +paths: + - 'python/**' + - '!python/Makefile' +

Re: [I] The payloadProps of AbstractRealtimeRecordReader is missing other configurations [hudi]

2024-07-25 Thread via GitHub
ailinzhou commented on issue #11685: URL: https://github.com/apache/hudi/issues/11685#issuecomment-2251845367 > I agree, we might need to generate the payload properties from the write config, can you fire a fix for it @ailinzhou ? cc @yihua for visibility. Sure, I can fix it. I'll

[jira] [Closed] (HUDI-7993) Support pruning and skipping with meta fields

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-7993. - Resolution: Fixed > Support pruning and skipping with meta fields >

[jira] [Updated] (HUDI-6191) Improve passing the debezium checkpoint values to start job from offset

2024-07-25 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6191: - Status: Open (was: In Progress) > Improve passing the debezium checkpoint values to start job from

[jira] [Closed] (HUDI-6191) Improve passing the debezium checkpoint values to start job from offset

2024-07-25 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6191. Resolution: Fixed Fixed via master branch: f54b9bb8dd7c77ba159f71feaca6f8475c15b535 > Improve passing the

(hudi) branch master updated: [HUDI-6191] Improve passing the debezium checkpoint values to start job from offset (#11686)

2024-07-25 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new f54b9bb8dd7 [HUDI-6191] Improve passing the

Re: [PR] [HUDI-6191] Improve passing the debezium checkpoint values to start job from offset [hudi]

2024-07-25 Thread via GitHub
danny0405 merged PR #11686: URL: https://github.com/apache/hudi/pull/11686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [SUPPORT] spark-sql write the table fail witch was created by flink-sql with HoodieHiveCatalog and the error is:Config conflict(key current value existing value): PreCombineKey: ts [hudi]

2024-07-25 Thread via GitHub
danny0405 commented on issue #11687: URL: https://github.com/apache/hudi/issues/11687#issuecomment-2251779686 @wardlican Did you config the precombine key already with option: `hoodie.datasource.write.precombine.field` ? -- This is an automated message from the Apache Git Service. To

Re: [PR] [HUDI-7989] Fix secondary index updates [hudi]

2024-07-25 Thread via GitHub
yihua commented on code in PR #11634: URL: https://github.com/apache/hudi/pull/11634#discussion_r1692359991 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -1097,7 +1096,7 @@ private HoodieData

Re: [PR] [HUDI-4625] Clean up KafkaOffsetGen: introduce retrying KafkaConsumer [hudi]

2024-07-25 Thread via GitHub
danny0405 commented on code in PR #11664: URL: https://github.com/apache/hudi/pull/11664#discussion_r1692359828 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KafkaSourceConfig.java: ## @@ -66,14 +66,6 @@ public class KafkaSourceConfig extends HoodieConfig {

Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]

2024-07-25 Thread via GitHub
yihua commented on code in PR #11579: URL: https://github.com/apache/hudi/pull/11579#discussion_r1692356320 ## hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java: ## @@ -332,7 +332,7 @@ public final class HoodieMetadataConfig extends HoodieConfig

Re: [PR] [HUDI-7996] Store partition type with partition fields in table configs [hudi]

2024-07-25 Thread via GitHub
danny0405 commented on code in PR #11638: URL: https://github.com/apache/hudi/pull/11638#discussion_r1692358034 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/util/SparkKeyGenUtils.scala: ## @@ -65,10 +74,15 @@ object SparkKeyGenUtils { // For

Re: [I] [SUPPORT] Migration partitionned table with complex key generator to 0.14.1 leads to duplicates when recordkey length =1 [hudi]

2024-07-25 Thread via GitHub
danny0405 commented on issue #10508: URL: https://github.com/apache/hudi/issues/10508#issuecomment-2251773525 > It would have been helpful to clearly highlight this in the documentation even though it was not intended considering how big the impact is. Sounds fair, would fire a fix

Re: [I] The payloadProps of AbstractRealtimeRecordReader is missing other configurations [hudi]

2024-07-25 Thread via GitHub
danny0405 commented on issue #11685: URL: https://github.com/apache/hudi/issues/11685#issuecomment-2251771675 I agree, we might need to generate the payload properties from the write config, can you fire a fix for it @ailinzhou ? cc @yihua for visibility. -- This is an automated message

Re: [I] [SUPPORT] Flink engine support RECORD_INDEX [hudi]

2024-07-25 Thread via GitHub
danny0405 commented on issue #11676: URL: https://github.com/apache/hudi/issues/11676#issuecomment-2251768794 @yongster Got it, we have support primary key filtering for bucket index for both Flink and Spark query engine, as for RLI, the data pruning on query side is still on the way,

Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]

2024-07-25 Thread via GitHub
yihua commented on PR #11579: URL: https://github.com/apache/hudi/pull/11579#issuecomment-2251769299 > I really feel we should cut down on the no of cols we generate stats out of the box. I have encountered OSS users give col stats a try and since it takes lot of time to populate col stats

Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]

2024-07-25 Thread via GitHub
wombatu-kun commented on issue #10644: URL: https://github.com/apache/hudi/issues/10644#issuecomment-2251767611 > @wombatu-kun can you take care of this? Yes of course, i'm already trying to figure out -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]

2024-07-25 Thread via GitHub
danny0405 commented on issue #10644: URL: https://github.com/apache/hudi/issues/10644#issuecomment-2251766311 @wombatu-kun can you take care of this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [HUDI-7580] Fix order of fields when records inserted out of order [hudi]

2024-07-25 Thread via GitHub
yihua commented on PR #11019: URL: https://github.com/apache/hudi/pull/11019#issuecomment-2251764603 @codope Is this PR still relevant? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [MINOR][DO NOT MERGE] Create release branch for version 1.0.0-beta2 [hudi]

2024-07-25 Thread via GitHub
yihua closed pull request #11558: [MINOR][DO NOT MERGE] Create release branch for version 1.0.0-beta2 URL: https://github.com/apache/hudi/pull/11558 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [MINOR][DO NOT MERGE] Create release branch for version 1.0.0-beta2 [hudi]

2024-07-25 Thread via GitHub
yihua commented on PR #11558: URL: https://github.com/apache/hudi/pull/11558#issuecomment-2251763031 Closing this as 1.0.0-beta2 is released. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Created] (HUDI-8027) Enable row writer for clustering

2024-07-25 Thread Lin Liu (Jira)
Lin Liu created HUDI-8027: - Summary: Enable row writer for clustering Key: HUDI-8027 URL: https://issues.apache.org/jira/browse/HUDI-8027 Project: Apache Hudi Issue Type: Bug Reporter:

Re: [PR] [HUDI-7919] docker demo spark3.5 support for IT [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11639: URL: https://github.com/apache/hudi/pull/11639#issuecomment-2251510381 ## CI report: * 0ccee2dd516c6a63e9eaf980ae3158daccb1c3c7 UNKNOWN * 3bbde49f851d7cd55e5c0ed0f9ba3cfc35f90ebf UNKNOWN * 10ca80806dc0d1bf854139039c01aa06faa382cc Azure:

Re: [PR] [HUDI-7919] docker demo spark3.5 support for IT [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11639: URL: https://github.com/apache/hudi/pull/11639#issuecomment-2251452192 ## CI report: * 0ccee2dd516c6a63e9eaf980ae3158daccb1c3c7 UNKNOWN * 5870acbeb83c028accc7530ba999cdc1e5653b9e Azure:

Re: [PR] [HUDI-7919] docker demo spark3.5 support for IT [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11639: URL: https://github.com/apache/hudi/pull/11639#issuecomment-2251441932 ## CI report: * 0ccee2dd516c6a63e9eaf980ae3158daccb1c3c7 UNKNOWN * 5870acbeb83c028accc7530ba999cdc1e5653b9e Azure:

Re: [PR] [HUDI-7919] docker demo spark3.5 support for IT [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11639: URL: https://github.com/apache/hudi/pull/11639#issuecomment-2251380638 ## CI report: * 0ccee2dd516c6a63e9eaf980ae3158daccb1c3c7 UNKNOWN * 5b2d65d061f86764ed5328c22a165a8a6077f22b Azure:

Re: [PR] [HUDI-7919] docker demo spark3.5 support for IT [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11639: URL: https://github.com/apache/hudi/pull/11639#issuecomment-2251368109 ## CI report: * 0ccee2dd516c6a63e9eaf980ae3158daccb1c3c7 UNKNOWN * 970c588713d8fc4ded73c7f1778cd3fe05e48507 Azure:

Re: [PR] [HUDI-7919] docker demo spark3.5 support for IT [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11639: URL: https://github.com/apache/hudi/pull/11639#issuecomment-2251355017 ## CI report: * 0ccee2dd516c6a63e9eaf980ae3158daccb1c3c7 UNKNOWN * 970c588713d8fc4ded73c7f1778cd3fe05e48507 Azure:

(hudi) branch master updated: [HUDI-7993] Apply record index if _hoodie_record_key meta field in filter (#11678)

2024-07-25 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 23a8b602475 [HUDI-7993] Apply record index if

Re: [PR] [HUDI-7993] Apply record index if _hoodie_record_key meta field in filter [hudi]

2024-07-25 Thread via GitHub
yihua merged PR #11678: URL: https://github.com/apache/hudi/pull/11678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-7919] docker demo spark3.5 support for IT [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11639: URL: https://github.com/apache/hudi/pull/11639#issuecomment-2251289443 ## CI report: * 1e51239d31cd48a4f9a96e8e401ed5b233924425 Azure:

Re: [PR] [HUDI-7919] docker demo spark3.5 support for IT [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11639: URL: https://github.com/apache/hudi/pull/11639#issuecomment-2251276322 ## CI report: * 1e51239d31cd48a4f9a96e8e401ed5b233924425 Azure:

Re: [PR] [HUDI-8003] Add hive overwrite payload [hudi]

2024-07-25 Thread via GitHub
jonvex commented on code in PR #11649: URL: https://github.com/apache/hudi/pull/11649#discussion_r169149 ## hudi-client/hudi-java-client/src/test/java/org/apache/hudi/hadoop/TestHoodieFileGroupReaderOnHive.java: ## @@ -0,0 +1,329 @@ +/* + * Licensed to the Apache Software

Re: [PR] [HUDI-8015] Fix Glue Meta Sync Failure on base path change [hudi]

2024-07-25 Thread via GitHub
CTTY commented on code in PR #11677: URL: https://github.com/apache/hudi/pull/11677#discussion_r1691991060 ## hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java: ## @@ -560,26 +560,11 @@ public void createOrReplaceTable(String tableName, //

Re: [PR] [HUDI-6453] Cascade Glue schema changes to partitions [hudi]

2024-07-25 Thread via GitHub
CTTY commented on PR #11670: URL: https://github.com/apache/hudi/pull/11670#issuecomment-2251212267 Thanks for helping continue the work! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Json schema decimal kafka [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11688: URL: https://github.com/apache/hudi/pull/11688#issuecomment-2251195121 ## CI report: * eafb13f42b391af09cd14e43de26da4cf29c0b0d Azure:

Re: [PR] [HUDI-7919] docker demo spark3.5 support for IT [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11639: URL: https://github.com/apache/hudi/pull/11639#issuecomment-2251194868 ## CI report: * 1e51239d31cd48a4f9a96e8e401ed5b233924425 Azure:

Re: [PR] [HUDI-8016] LastSyncedTime is not updated for Snapshot table in Glue Sync [hudi]

2024-07-25 Thread via GitHub
CTTY commented on PR #11681: URL: https://github.com/apache/hudi/pull/11681#issuecomment-2251197716 Hi @vamsikarnika, could you paste the stacktrace here and provide a bit more context about the issue? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [HUDI-7919] docker demo spark3.5 support for IT [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11639: URL: https://github.com/apache/hudi/pull/11639#issuecomment-2251178592 ## CI report: * 43f383f8b3c2150a7cfd2843f04d44d4ef10adde Azure:

Re: [PR] [MINOR][DO NOT MERGE] Turn on data skipping and native hfile reader by default [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11620: URL: https://github.com/apache/hudi/pull/11620#issuecomment-2251163842 ## CI report: * 3366ddc64631a54d9dc0f3ee482a55fd4c6dd24a Azure:

Re: [PR] Json schema decimal kafka [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11688: URL: https://github.com/apache/hudi/pull/11688#issuecomment-2251081905 ## CI report: * eafb13f42b391af09cd14e43de26da4cf29c0b0d Azure:

Re: [PR] [HUDI-7403] Support Filter/Transformer to Hudi Exporter Utility [hudi]

2024-07-25 Thread via GitHub
soumilshah1995 commented on PR #11509: URL: https://github.com/apache/hudi/pull/11509#issuecomment-2251082324 Hey @danny0405 im seeing error while using it https://github.com/apache/hudi/issues/10644 -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [HUDI-7993] Apply record index if _hoodie_record_key meta field in filter [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11678: URL: https://github.com/apache/hudi/pull/11678#issuecomment-2251081782 ## CI report: * de9e2e42546f2c6e43a5a6fa149c402cba06e7c2 UNKNOWN * a6f849e830a81ff545ea7a6a157e48891cdcc02e Azure:

Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]

2024-07-25 Thread via GitHub
yihua commented on code in PR #11597: URL: https://github.com/apache/hudi/pull/11597#discussion_r1691884566 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java: ## @@ -288,19 +277,19 @@ private HoodieMetadataPayload(String key, int type, Map

Re: [I] [FeatureRequest] Inquiry Regarding Hudi Exporter with SQL Transformer for Data Filtering [hudi]

2024-07-25 Thread via GitHub
soumilshah1995 commented on issue #10644: URL: https://github.com/apache/hudi/issues/10644#issuecomment-2251065010 Hi there I saw that this ticket was completed and I was trying out this functionality Docs ``` xport to json or parquet dataset with transformation/filtering

Re: [PR] Json schema decimal kafka [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11688: URL: https://github.com/apache/hudi/pull/11688#issuecomment-2251065218 ## CI report: * eafb13f42b391af09cd14e43de26da4cf29c0b0d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]

2024-07-25 Thread via GitHub
yihua commented on code in PR #11597: URL: https://github.com/apache/hudi/pull/11597#discussion_r1691875001 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java: ## @@ -211,7 +200,7 @@ public HoodieMetadataPayload(Option recordOpt) { key =

Re: [PR] [HUDI-7993] Apply record index if _hoodie_record_key meta field in filter [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11678: URL: https://github.com/apache/hudi/pull/11678#issuecomment-2251050440 ## CI report: * de9e2e42546f2c6e43a5a6fa149c402cba06e7c2 UNKNOWN * 7d6d0c89f9c34bb0c50844d8e9209b774d12b3ee Azure:

[jira] [Updated] (HUDI-8026) Test multiple indexes creation and updates together

2024-07-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-8026: - Labels: pull-request-available (was: ) > Test multiple indexes creation and updates together >

[jira] [Updated] (HUDI-8024) Test index updates and rollback

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8024: -- Status: In Progress (was: Open) > Test index updates and rollback >

[jira] [Updated] (HUDI-8026) Test multiple indexes creation and updates together

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8026: -- Status: In Progress (was: Open) > Test multiple indexes creation and updates together >

[PR] [WIP][HUDI-8026] Add tests with mutliple indexes [hudi]

2024-07-25 Thread via GitHub
codope opened a new pull request, #11689: URL: https://github.com/apache/hudi/pull/11689 ### Change Logs Add secondary index tests with partiion stats index also enabled and verufy data skipping ### Impact Enahnce test coverage ### Risk level (write none, low

[PR] Json schema decimal kafka [hudi]

2024-07-25 Thread via GitHub
jonvex opened a new pull request, #11688: URL: https://github.com/apache/hudi/pull/11688 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any

[jira] [Closed] (HUDI-8013) Test Plan for multi-modal index

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-8013. - Resolution: Done > Test Plan for multi-modal index > --- > >

[jira] [Updated] (HUDI-8013) Test Plan for multi-modal index

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8013: -- Status: Patch Available (was: In Progress) > Test Plan for multi-modal index >

[jira] [Updated] (HUDI-8013) Test Plan for multi-modal index

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8013: -- Status: In Progress (was: Open) > Test Plan for multi-modal index > --- >

[jira] [Created] (HUDI-8026) Test multiple indexes creation and updates together

2024-07-25 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-8026: - Summary: Test multiple indexes creation and updates together Key: HUDI-8026 URL: https://issues.apache.org/jira/browse/HUDI-8026 Project: Apache Hudi Issue Type:

Re: [PR] [HUDI-7993] Apply record index if _hoodie_record_key meta field in filter [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11678: URL: https://github.com/apache/hudi/pull/11678#issuecomment-2250973692 ## CI report: * de9e2e42546f2c6e43a5a6fa149c402cba06e7c2 UNKNOWN * 7d6d0c89f9c34bb0c50844d8e9209b774d12b3ee Azure:

Re: [PR] [MINOR][DO NOT MERGE] Turn on data skipping and native hfile reader by default [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11620: URL: https://github.com/apache/hudi/pull/11620#issuecomment-2250973408 ## CI report: * b856cf8061883c72bff5551894d871ea6bae62a3 Azure:

Re: [PR] [MINOR][DO NOT MERGE] Turn on data skipping and native hfile reader by default [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11620: URL: https://github.com/apache/hudi/pull/11620#issuecomment-2250952573 ## CI report: * b856cf8061883c72bff5551894d871ea6bae62a3 Azure:

[jira] [Updated] (HUDI-8024) Test index updates and rollback

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8024: -- Story Points: 3 (was: 4) > Test index updates and rollback > > >

Re: [PR] [HUDI-7993] Apply record index if _hoodie_record_key meta field in filter [hudi]

2024-07-25 Thread via GitHub
yihua commented on code in PR #11678: URL: https://github.com/apache/hudi/pull/11678#discussion_r1691800246 ## hudi-spark-datasource/hudi-spark-common/src/test/scala/org/apache/hudi/TestRecordLevelIndexSupport.scala: ## @@ -43,46 +48,62 @@ class TestRecordLevelIndexSupport {

[jira] [Created] (HUDI-8024) Test index updates and rollback

2024-07-25 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-8024: - Summary: Test index updates and rollback Key: HUDI-8024 URL: https://issues.apache.org/jira/browse/HUDI-8024 Project: Apache Hudi Issue Type: Test

[jira] [Assigned] (HUDI-8024) Test index updates and rollback

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-8024: - Assignee: Sagar Sumit > Test index updates and rollback > > >

[jira] [Updated] (HUDI-8024) Test index updates and rollback

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8024: -- Story Points: 4 > Test index updates and rollback > > >

[jira] [Updated] (HUDI-8023) Add multi-writer tests for indexes

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8023: -- Story Points: 3 (was: 4) > Add multi-writer tests for indexes > -- > >

[jira] [Created] (HUDI-8025) Test all indexes with compaction and cleaning

2024-07-25 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-8025: - Summary: Test all indexes with compaction and cleaning Key: HUDI-8025 URL: https://issues.apache.org/jira/browse/HUDI-8025 Project: Apache Hudi Issue Type: Test

[jira] [Updated] (HUDI-8024) Test index updates and rollback

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8024: -- Fix Version/s: 1.0.0 > Test index updates and rollback > > >

[jira] [Created] (HUDI-8023) Add multi-writer tests for indexes

2024-07-25 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-8023: - Summary: Add multi-writer tests for indexes Key: HUDI-8023 URL: https://issues.apache.org/jira/browse/HUDI-8023 Project: Apache Hudi Issue Type: Test

[jira] [Created] (HUDI-8022) All positive index tests should validate data skipping

2024-07-25 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-8022: - Summary: All positive index tests should validate data skipping Key: HUDI-8022 URL: https://issues.apache.org/jira/browse/HUDI-8022 Project: Apache Hudi Issue

Re: [PR] [HUDI-7993] Apply record index if _hoodie_record_key meta field in filter [hudi]

2024-07-25 Thread via GitHub
yihua commented on code in PR #11678: URL: https://github.com/apache/hudi/pull/11678#discussion_r1691785701 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala: ## @@ -70,45 +77,45 @@ class TestRecordLevelIndexWithSQL

[jira] [Created] (HUDI-8021) Cover Clustering in TestUpdateTable

2024-07-25 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-8021: - Summary: Cover Clustering in TestUpdateTable Key: HUDI-8021 URL: https://issues.apache.org/jira/browse/HUDI-8021 Project: Apache Hudi Issue Type: Test

[jira] [Created] (HUDI-8020) Test Compaction with deletes in SQL

2024-07-25 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-8020: - Summary: Test Compaction with deletes in SQL Key: HUDI-8020 URL: https://issues.apache.org/jira/browse/HUDI-8020 Project: Apache Hudi Issue Type: Test

Re: [I] [SUPPORT] The cleaning service takes a long time [hudi]

2024-07-25 Thread via GitHub
nb commented on issue #11680: URL: https://github.com/apache/hudi/issues/11680#issuecomment-2250883054 @ad1happy2go ![1](https://github.com/user-attachments/assets/cf55ba1c-182e-4474-a454-914059e6d4e6) Do you want this hoodie timeline information? -- This is an automated

[jira] [Updated] (HUDI-8019) Merge some redundant SQL tests

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8019: -- Description: In TestSpark3DDL, merge some redundant tests such as double to decimal, string to

[jira] [Updated] (HUDI-8019) Merge some redundant SQL tests

2024-07-25 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-8019: -- Summary: Merge some redundant SQL tests (was: Merge some redundant tests in TestSpark3DDL) > Merge

Re: [I] [SUPPORT] The cleaning service takes a long time [hudi]

2024-07-25 Thread via GitHub
nb commented on issue #11680: URL: https://github.com/apache/hudi/issues/11680#issuecomment-2250867126 @ad1happy2go The data source is Apache Kudu, which incrementally writes data to the Hudi table through Spark here is my hudi configs hoodie.metadata.record.index.enable=true

[jira] [Created] (HUDI-8019) Merge some redundant tests in TestSpark3DDL

2024-07-25 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-8019: - Summary: Merge some redundant tests in TestSpark3DDL Key: HUDI-8019 URL: https://issues.apache.org/jira/browse/HUDI-8019 Project: Apache Hudi Issue Type: Test

[jira] [Created] (HUDI-8018) Parameterize most SQL tests for both table types

2024-07-25 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-8018: - Summary: Parameterize most SQL tests for both table types Key: HUDI-8018 URL: https://issues.apache.org/jira/browse/HUDI-8018 Project: Apache Hudi Issue Type:

Re: [I] Hudi job hangs forever [hudi]

2024-07-25 Thread via GitHub
keerthiskating commented on issue #11540: URL: https://github.com/apache/hudi/issues/11540#issuecomment-2250767531 I am facing similar issue. I am using Glue 4.0 with Hudi 0.14. The very first run when the table does not exist in Glue catalog works fine. However, over time, the glue job

(hudi) branch master updated: [HUDI-8005] Add a new DynamoDB based lock provider with auto-generated partition key (#11667)

2024-07-25 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 7232d2461f4 [HUDI-8005] Add a new DynamoDB based

Re: [PR] [HUDI-8005] Add a new DynamoDB based lock provider with auto-generated partition key [hudi]

2024-07-25 Thread via GitHub
yihua merged PR #11667: URL: https://github.com/apache/hudi/pull/11667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-8003] Add hive overwrite payload [hudi]

2024-07-25 Thread via GitHub
jonvex commented on code in PR #11649: URL: https://github.com/apache/hudi/pull/11649#discussion_r1691685510 ## hudi-client/hudi-java-client/src/test/java/org/apache/hudi/hadoop/TestHoodieFileGroupReaderOnHive.java: ## @@ -0,0 +1,329 @@ +/* + * Licensed to the Apache Software

[jira] [Created] (HUDI-8017) Merge mode / write payload requires redundant configuration to work.

2024-07-25 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-8017: - Summary: Merge mode / write payload requires redundant configuration to work. Key: HUDI-8017 URL: https://issues.apache.org/jira/browse/HUDI-8017 Project: Apache

Re: [I] [SUPPORT] RFC 63 Functional Index Hudi 0.1.0-beta [hudi]

2024-07-25 Thread via GitHub
soumilshah1995 commented on issue #10110: URL: https://github.com/apache/hudi/issues/10110#issuecomment-2250710861 just updating this thread I did small test # Before Index Creation ``` spark.read.format("hudi") \ .option("hoodie.enable.data.skipping", "true") \

Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11597: URL: https://github.com/apache/hudi/pull/11597#issuecomment-2250703652 ## CI report: * c4d4081b1d60551ae60f57f10163d0d2a63f0378 Azure:

Re: [PR] [HUDI-8005] Add new lock provider implementation [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11667: URL: https://github.com/apache/hudi/pull/11667#issuecomment-2250704416 ## CI report: * 87e7dd1ce50e523825faefcdb743c58cafd0799d Azure:

Re: [I] [SUPPORT] The cleaning service takes a long time [hudi]

2024-07-25 Thread via GitHub
ad1happy2go commented on issue #11680: URL: https://github.com/apache/hudi/issues/11680#issuecomment-2250681560 @nb I dont think cleaning can cause that. Are you saying when you stop the job and rerun , it again takes ~24 mins. What is the source of the data. Can you let us know the

Re: [PR] ci: run rust-tests on windows [hudi-rs]

2024-07-25 Thread via GitHub
crrow commented on code in PR #92: URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1691583734 ## .github/workflows/ci.yml: ## @@ -44,14 +44,16 @@ jobs: strategy: fail-fast: false matrix: -# TODO: add windows which does not support container

Re: [PR] [HUDI-8005] Add new lock provider implementation [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11667: URL: https://github.com/apache/hudi/pull/11667#issuecomment-2250509756 ## CI report: * 87e7dd1ce50e523825faefcdb743c58cafd0799d Azure:

Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11597: URL: https://github.com/apache/hudi/pull/11597#issuecomment-2250481596 ## CI report: * ec339e004d71a16e0b038bc7008385e1005f2367 Azure:

Re: [PR] [HUDI-8005] Add new lock provider implementation [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11667: URL: https://github.com/apache/hudi/pull/11667#issuecomment-2250482220 ## CI report: * 87e7dd1ce50e523825faefcdb743c58cafd0799d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7993] Apply record index if _hoodie_record_key meta field in filter [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11678: URL: https://github.com/apache/hudi/pull/11678#issuecomment-2250482494 ## CI report: * de9e2e42546f2c6e43a5a6fa149c402cba06e7c2 UNKNOWN * 7d6d0c89f9c34bb0c50844d8e9209b774d12b3ee Azure:

Re: [PR] [HUDI-7996] Store partition type with partition fields in table configs [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11638: URL: https://github.com/apache/hudi/pull/11638#issuecomment-2250373780 ## CI report: * 2ff72d788c843f1ed425e987dc0c87dfd872c9b3 Azure:

Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]

2024-07-25 Thread via GitHub
hudi-bot commented on PR #11597: URL: https://github.com/apache/hudi/pull/11597#issuecomment-2250373412 ## CI report: * ec339e004d71a16e0b038bc7008385e1005f2367 Azure:

Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]

2024-07-25 Thread via GitHub
codope commented on code in PR #11597: URL: https://github.com/apache/hudi/pull/11597#discussion_r1691467321 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java: ## @@ -211,7 +200,7 @@ public HoodieMetadataPayload(Option recordOpt) { key =

Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]

2024-07-25 Thread via GitHub
codope commented on code in PR #11597: URL: https://github.com/apache/hudi/pull/11597#discussion_r1691466848 ## hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java: ## @@ -137,6 +148,10 @@ public String getFileIdPrefix() { return fileIdPrefix;

  1   2   >