Re: [PR] [HUDI-7704] Unify test client storage classes with duplicate code [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11152: URL: https://github.com/apache/hudi/pull/11152#issuecomment-2105636243 ## CI report: * a5daf71906886e6d8da62abdf2decae1e20b09ef Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7704] Unify test client storage classes with duplicate code [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11152: URL: https://github.com/apache/hudi/pull/11152#issuecomment-2105638313 ## CI report: * a5daf71906886e6d8da62abdf2decae1e20b09ef Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [SUPPORT] spark stuctrued streaming failed to update MDT metadata [hudi]

2024-05-11 Thread via GitHub
xicm commented on issue #10891: URL: https://github.com/apache/hudi/issues/10891#issuecomment-2105645522 https://hudi.apache.org/docs/metadata#deployment-model-b-single-writer-with-async-table-services -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] [SUPPORT] spark stuctrued streaming failed to update MDT metadata [hudi]

2024-05-11 Thread via GitHub
xicm commented on issue #10891: URL: https://github.com/apache/hudi/issues/10891#issuecomment-2105647756 Maybe we should set default value of hoodie.datasource.compaction.async.enable to false. It's confusing to user that single writer needs a lock by default. @danny0405 -- This is

Re: [I] [SUPPORT] spark stuctrued streaming failed to update MDT metadata [hudi]

2024-05-11 Thread via GitHub
xicm closed issue #10891: [SUPPORT] spark stuctrued streaming failed to update MDT metadata URL: https://github.com/apache/hudi/issues/10891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] [HUDI-7704] Unify test client storage classes with duplicate code [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11152: URL: https://github.com/apache/hudi/pull/11152#issuecomment-2105655112 ## CI report: * d733b14808e91231b36d35ecd134b7802c3f8d33 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [I] [SUPPORT] IllegalArgumentException at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:33) [hudi]

2024-05-11 Thread via GitHub
xicm commented on issue #10906: URL: https://github.com/apache/hudi/issues/10906#issuecomment-2105655345 https://hudi.apache.org/docs/metadata#deployment-model-b-single-writer-with-async-table-services Lock is needed for asycn table service with MDT. -- This is an automated message

Re: [PR] [HUDI-7704] Unify test client storage classes with duplicate code [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11152: URL: https://github.com/apache/hudi/pull/11152#issuecomment-2105656914 ## CI report: * d733b14808e91231b36d35ecd134b7802c3f8d33 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7704] Unify test client storage classes with duplicate code [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11152: URL: https://github.com/apache/hudi/pull/11152#issuecomment-2105669259 ## CI report: * d733b14808e91231b36d35ecd134b7802c3f8d33 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7704] Unify test client storage classes with duplicate code [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11152: URL: https://github.com/apache/hudi/pull/11152#issuecomment-2105687296 ## CI report: * c5add8c5061d801c1c1ab0369f18954fa6b54d91 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11070: URL: https://github.com/apache/hudi/pull/11070#issuecomment-2105732770 ## CI report: * 86265f6be7c6fbacf53ed76a7b60b2b64d484409 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11070: URL: https://github.com/apache/hudi/pull/11070#issuecomment-2105735707 ## CI report: * 86265f6be7c6fbacf53ed76a7b60b2b64d484409 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

[I] [SUPPORT] Benchmarking Hudi Simple Index and Record level indexing for COW [hudi]

2024-05-11 Thread via GitHub
bibhu107 opened a new issue, #11194: URL: https://github.com/apache/hudi/issues/11194 We request the community to **Benchmark Record Level Indexing (RLI) with Simple Indexing**. The blog at https://hudi.apache.org/blog/2023/11/01/record-level-index/ provides a great comparison between

Re: [PR] [HUDI-7704] Unify test client storage classes with duplicate code [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11152: URL: https://github.com/apache/hudi/pull/11152#issuecomment-2105746760 ## CI report: * c5add8c5061d801c1c1ab0369f18954fa6b54d91 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7704] Unify test client storage classes with duplicate code [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11152: URL: https://github.com/apache/hudi/pull/11152#issuecomment-2105755951 ## CI report: * c5add8c5061d801c1c1ab0369f18954fa6b54d91 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-4732] Add support for confluent schema registry with proto [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11070: URL: https://github.com/apache/hudi/pull/11070#issuecomment-2105876784 ## CI report: * 36cecc155c8cf9fbec29f895eea650eacf0b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7743] Improve StoragePath usages [hudi]

2024-05-11 Thread via GitHub
jonvex commented on code in PR #11189: URL: https://github.com/apache/hudi/pull/11189#discussion_r1597457778 ## hudi-cli/src/main/java/org/apache/hudi/cli/commands/RepairsCommand.java: ## @@ -123,7 +121,7 @@ public String addPartitionMeta( client.getActiveTimeline().ge

[jira] [Created] (HUDI-7747) In MetaClient remove getBasePathV2() and return StoragePath from getBasePath()

2024-05-11 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-7747: - Summary: In MetaClient remove getBasePathV2() and return StoragePath from getBasePath() Key: HUDI-7747 URL: https://issues.apache.org/jira/browse/HUDI-7747 Project:

Re: [PR] [HUDI-7704] Unify test client storage classes with duplicate code [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11152: URL: https://github.com/apache/hudi/pull/11152#issuecomment-2105911476 ## CI report: * 0b6a222addb8158cd6981f22bfd131f3fb176939 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7743] Improve StoragePath usages [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11189: URL: https://github.com/apache/hudi/pull/11189#issuecomment-2105911836 ## CI report: * 975a7d92617080bb4c32e832796e8d13cd8d9857 UNKNOWN * 76ee9ca6a701a2fcaa70fce9aae46864486c8c45 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7743] Improve StoragePath usages [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11189: URL: https://github.com/apache/hudi/pull/11189#issuecomment-2105934253 ## CI report: * 975a7d92617080bb4c32e832796e8d13cd8d9857 UNKNOWN * fc5e90ed51e7c4151832c3a6c756d4b9ef93131f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-7743] Improve StoragePath usages [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11189: URL: https://github.com/apache/hudi/pull/11189#issuecomment-2105948149 ## CI report: * fc5e90ed51e7c4151832c3a6c756d4b9ef93131f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7743] Improve StoragePath usages [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11189: URL: https://github.com/apache/hudi/pull/11189#issuecomment-2105950146 ## CI report: * fc5e90ed51e7c4151832c3a6c756d4b9ef93131f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]

2024-05-11 Thread via GitHub
vinishjail97 commented on PR #10872: URL: https://github.com/apache/hudi/pull/10872#issuecomment-2105950888 > hey @vinishjail97 : can you attach the memory profileing you did before and after this patch. and rebase w/ master. we are good to go 15th March: Basic OOM Test (Consume 2

Re: [PR] [HUDI-7745] Move Hadoop-dependent util methods to hudi-hadoop-common module [hudi]

2024-05-11 Thread via GitHub
jonvex merged PR #11193: URL: https://github.com/apache/hudi/pull/11193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

(hudi) branch master updated (49072d1e2e7 -> 61f54a0dcc2)

2024-05-11 Thread jonvex
This is an automated email from the ASF dual-hosted git repository. jonvex pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 49072d1e2e7 [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPart

[I] [SUPPORT] Archival support for metadata table of Record level indexing [hudi]

2024-05-11 Thread via GitHub
bibhu107 opened a new issue, #11195: URL: https://github.com/apache/hudi/issues/11195 Hi Community, I am seeking guidance on implementing archival policies for the Metadata table in Hudi's record-level indexing (RLI). We are not concerned with retaining data older than three years, a

Re: [PR] [HUDI-7743] Improve StoragePath usages [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11189: URL: https://github.com/apache/hudi/pull/11189#issuecomment-2105965828 ## CI report: * fc5e90ed51e7c4151832c3a6c756d4b9ef93131f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7744] Introduce IOFactory and a config to set the factory [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11192: URL: https://github.com/apache/hudi/pull/11192#issuecomment-2105965841 ## CI report: * 81806555cd6c82297f2ff34b81466e653b483a61 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7743] Improve StoragePath usages [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11189: URL: https://github.com/apache/hudi/pull/11189#issuecomment-2105968110 ## CI report: * fc5e90ed51e7c4151832c3a6c756d4b9ef93131f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7744] Introduce IOFactory and a config to set the factory [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11192: URL: https://github.com/apache/hudi/pull/11192#issuecomment-2105968180 ## CI report: * 81806555cd6c82297f2ff34b81466e653b483a61 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7743] Improve StoragePath usages [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11189: URL: https://github.com/apache/hudi/pull/11189#issuecomment-2105984164 ## CI report: * 511e55b8d042e8db674b48b203f3bf9b8f52ad6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7744] Introduce IOFactory and a config to set the factory [hudi]

2024-05-11 Thread via GitHub
hudi-bot commented on PR #11192: URL: https://github.com/apache/hudi/pull/11192#issuecomment-2105992231 ## CI report: * cb4f0d22f065fcbb222def1d4bbc8a8f822ef25d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23

Re: [PR] [HUDI-7704] Unify test client storage classes with duplicate code [hudi]

2024-05-11 Thread via GitHub
danny0405 commented on PR #11152: URL: https://github.com/apache/hudi/pull/11152#issuecomment-2106088474 cc @the-other-tim-brown for the code reviewing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] [SUPPORT] Archival support for metadata table of Record level indexing [hudi]

2024-05-11 Thread via GitHub
danny0405 commented on issue #11195: URL: https://github.com/apache/hudi/issues/11195#issuecomment-2106088949 > I wanted to inquire if there are any plans to introduce TTL or Archival support for the Hudi metadata table's record keys. Record level ttl is a very tipical request for reg

Re: [I] [SUPPORT] Benchmarking Hudi Simple Index and Record level indexing for COW [hudi]

2024-05-11 Thread via GitHub
ad1happy2go commented on issue #11194: URL: https://github.com/apache/hudi/issues/11194#issuecomment-2106113688 @bibhu107 RLI is mainly designed to use as GLOBAL INDEX. So in your use case you may not need to create a custom partition column to improve performance. RLI will work good, as it

[I] Adding New Configuration To Support ZSTD Level [hudi]

2024-05-11 Thread via GitHub
Amar1404 opened a new issue, #11196: URL: https://github.com/apache/hudi/issues/11196 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-subsc