Re: [PR] [HUDI-8110] Throw an error for time travel query on MDT [hudi]

2024-08-25 Thread via GitHub
lokeshj1703 commented on PR #11819: URL: https://github.com/apache/hudi/pull/11819#issuecomment-2309447896 @CTTY I think the time travel queries on MDT should be executed mainly for debugging purposes. The hudi components do not use time travel query on MDT. -- This is an automated messag

Re: [PR] [HUDI-8110] Throw an error for time travel query on MDT [hudi]

2024-08-25 Thread via GitHub
hudi-bot commented on PR #11819: URL: https://github.com/apache/hudi/pull/11819#issuecomment-2309445548 ## CI report: * 5c24340a813bb2ad749f9b602a707200cfb1e0c5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=161)

Re: [PR] [HUDI-7711] Fix MultiTableStreamer can deal with path of properties files [hudi]

2024-08-25 Thread via GitHub
lokeshj1703 commented on code in PR #11149: URL: https://github.com/apache/hudi/pull/11149#discussion_r1730740068 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieMultiTableStreamer.java: ## @@ -430,6 +432,99 @@ public static class Config implements Seria

Re: [PR] [HUDI-7711] Fix MultiTableStreamer can deal with path of properties files [hudi]

2024-08-25 Thread via GitHub
lokeshj1703 commented on code in PR #11149: URL: https://github.com/apache/hudi/pull/11149#discussion_r1730720592 ## hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/TestStreamSyncUnitTests.java: ## @@ -160,6 +165,45 @@ void testGetCheckpointToResume(HoodieStreame

Re: [PR] [MINOR] Add backward compatibility test for HoodieTableConfig partition field APIs [hudi]

2024-08-25 Thread via GitHub
hudi-bot commented on PR #11824: URL: https://github.com/apache/hudi/pull/11824#issuecomment-2309365529 ## CI report: * 0640fa3d9bbadd79cf4d3bc025f35cf679bf7280 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=162)

Re: [PR] [MINOR] Add backward compatibility test for HoodieTableConfig partition field APIs [hudi]

2024-08-25 Thread via GitHub
hudi-bot commented on PR #11824: URL: https://github.com/apache/hudi/pull/11824#issuecomment-2309363651 ## CI report: * 0640fa3d9bbadd79cf4d3bc025f35cf679bf7280 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7970] Move methods from HoodieTableConfigUtils to HoodieTableConfig [hudi]

2024-08-25 Thread via GitHub
lokeshj1703 commented on code in PR #11601: URL: https://github.com/apache/hudi/pull/11601#discussion_r1730694520 ## hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java: ## @@ -518,6 +518,58 @@ public static boolean validateChecksum(Properties props) {

[PR] [MINOR] Add backward compatibility test for HoodieTableConfig partition field APIs [hudi]

2024-08-25 Thread via GitHub
lokeshj1703 opened a new pull request, #11824: URL: https://github.com/apache/hudi/pull/11824 ### Change Logs Adds a test to ensure the backward compatibility is maintained for the partiton field APIs. Partition fields in table config now store the partition field type as well which

Re: [PR] [HUDI-8110] Throw an error for time travel query on MDT [hudi]

2024-08-25 Thread via GitHub
hudi-bot commented on PR #11819: URL: https://github.com/apache/hudi/pull/11819#issuecomment-2309356801 ## CI report: * af84acaf956239bddc450933d5ef94cc6e6fdf1e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=157)

Re: [PR] [HUDI-8110] Throw an error for time travel query on MDT [hudi]

2024-08-25 Thread via GitHub
hudi-bot commented on PR #11819: URL: https://github.com/apache/hudi/pull/11819#issuecomment-2309355306 ## CI report: * af84acaf956239bddc450933d5ef94cc6e6fdf1e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=157)

Re: [PR] feat: support partition prune api [hudi-rs]

2024-08-25 Thread via GitHub
KnightChess commented on PR #119: URL: https://github.com/apache/hudi-rs/pull/119#issuecomment-2309312849 @xushiyan cc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[jira] [Created] (HUDI-8121) Flink implements the skip merge queries using the file group reader

2024-08-25 Thread Danny Chen (Jira)
Danny Chen created HUDI-8121: Summary: Flink implements the skip merge queries using the file group reader Key: HUDI-8121 URL: https://issues.apache.org/jira/browse/HUDI-8121 Project: Apache Hudi

[jira] [Created] (HUDI-8120) Flink implements the read optimized queries using the file group reader

2024-08-25 Thread Danny Chen (Jira)
Danny Chen created HUDI-8120: Summary: Flink implements the read optimized queries using the file group reader Key: HUDI-8120 URL: https://issues.apache.org/jira/browse/HUDI-8120 Project: Apache Hudi

[jira] [Created] (HUDI-8119) Flink implements the snapshot queries using the file group reader

2024-08-25 Thread Danny Chen (Jira)
Danny Chen created HUDI-8119: Summary: Flink implements the snapshot queries using the file group reader Key: HUDI-8119 URL: https://issues.apache.org/jira/browse/HUDI-8119 Project: Apache Hudi

[jira] [Created] (HUDI-8117) Implement the HoodieFlinkRecord akka to the HoodieSparkRecord

2024-08-25 Thread Danny Chen (Jira)
Danny Chen created HUDI-8117: Summary: Implement the HoodieFlinkRecord akka to the HoodieSparkRecord Key: HUDI-8117 URL: https://issues.apache.org/jira/browse/HUDI-8117 Project: Apache Hudi Issu

[jira] [Created] (HUDI-8118) Implement the Flink specific FileGroupReader with the HoodieFlinkRecord

2024-08-25 Thread Danny Chen (Jira)
Danny Chen created HUDI-8118: Summary: Implement the Flink specific FileGroupReader with the HoodieFlinkRecord Key: HUDI-8118 URL: https://issues.apache.org/jira/browse/HUDI-8118 Project: Apache Hudi

[jira] [Updated] (HUDI-6788) Integrate FileGroupReader with MergeOnReadInputFormat for Flink

2024-08-25 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6788: - Description: The existing MergeOnReadInputFormat implements different iterators for all kinds of read mor

[jira] [Updated] (HUDI-6788) Integrate FileGroupReader with MergeOnReadInputFormat for Flink

2024-08-25 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6788: - Description: The existing  > Integrate FileGroupReader with MergeOnReadInputFormat for Flink > ---

Re: [PR] [HUDI-1749] Clean/Compaction/Rollback command maybe never exit when operation fail [hudi]

2024-08-25 Thread via GitHub
codecov-commenter commented on PR #2752: URL: https://github.com/apache/hudi/pull/2752#issuecomment-2309000407 ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/2752?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_te

Re: [I] [SUPPORT] [hudi]

2024-08-25 Thread via GitHub
cbomgit commented on issue #11821: URL: https://github.com/apache/hudi/issues/11821#issuecomment-2308978687 Thanks. Switching to SIMPLE index doesn't seem to help. Job fails at the following job and stage: ![Screenshot 2024-08-25 at 2 08 33  PM](https://github.com/user-attachments/ass

Re: [PR] [HUDI-7696][Draft] Refactoring functions in HoodieTableMetadataUtil [hudi]

2024-08-25 Thread via GitHub
hudi-bot commented on PR #11823: URL: https://github.com/apache/hudi/pull/11823#issuecomment-2308943689 ## CI report: * d0afece207d5b2bdedf55d0c63b926089560ea20 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=160)

Re: [PR] [HUDI-7696][Draft] Refactoring functions in HoodieTableMetadataUtil [hudi]

2024-08-25 Thread via GitHub
hudi-bot commented on PR #11823: URL: https://github.com/apache/hudi/pull/11823#issuecomment-2308922768 ## CI report: * d0afece207d5b2bdedf55d0c63b926089560ea20 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=160)

Re: [PR] [HUDI-7696][Draft] Refactoring functions in HoodieTableMetadataUtil [hudi]

2024-08-25 Thread via GitHub
hudi-bot commented on PR #11823: URL: https://github.com/apache/hudi/pull/11823#issuecomment-2308922258 ## CI report: * d0afece207d5b2bdedf55d0c63b926089560ea20 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Updated] (HUDI-7696) Consolidate convertFilesToPartitionStatsRecords and convertMetadataToPartitionStatsRecords

2024-08-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7696: - Labels: pull-request-available (was: ) > Consolidate convertFilesToPartitionStatsRecords and > c

[PR] [HUDI-7696][Draft] Refactoring functions in HoodieTableMetadataUtil [hudi]

2024-08-25 Thread via GitHub
bibhu107 opened a new pull request, #11823: URL: https://github.com/apache/hudi/pull/11823 ### Change Logs Two functions in `HoodieTableMetadataUtil`, `convertFilesToPartitionStatsRecords` and `convertMetadataToPartitionStatsRecords`, are being evaluated for unification. These fu

Re: [PR] chore: configure pre-commit hooks for code formatting [hudi-rs]

2024-08-25 Thread via GitHub
muyihao commented on code in PR #103: URL: https://github.com/apache/hudi-rs/pull/103#discussion_r1730390321 ## .pre-commit-config.yaml: ## @@ -0,0 +1,31 @@ +repos: + - repo: https://github.com/pre-commit/pre-commit-hooks +rev: v4.6.0 +hooks: + - id: trailing-white

Re: [PR] chore: configure pre-commit hooks for code formatting [hudi-rs]

2024-08-25 Thread via GitHub
muyihao commented on code in PR #103: URL: https://github.com/apache/hudi-rs/pull/103#discussion_r1730386898 ## python/Makefile: ## @@ -30,6 +30,8 @@ setup-venv: ## Setup the virtualenv setup: ## Setup the requirements $(info --- Setup dependencies ---) pip inst

Re: [I] [SUPPORT] Hudi very slow performance when streaming high number of partitioned data from kinesis. [hudi]

2024-08-25 Thread via GitHub
ad1happy2go commented on issue #11820: URL: https://github.com/apache/hudi/issues/11820#issuecomment-2308871247 @epsilon-akshay Can you post the screenshots of spark UI. Which stage is taking time. -- This is an automated message from the Apache Git Service. To respond to the message, ple

[PR] feat: support partition prune api [hudi-rs]

2024-08-25 Thread via GitHub
KnightChess opened a new pull request, #119: URL: https://github.com/apache/hudi-rs/pull/119 ## Description #47 support hudi internal filter, engine like datafusion or other need cover it expression to hudi partition filter. something like this code, ```rust let filter_

Re: [PR] [Minor] Removing unused function convertMetadataToRecords [hudi]

2024-08-25 Thread via GitHub
hudi-bot commented on PR #11822: URL: https://github.com/apache/hudi/pull/11822#issuecomment-2308718036 ## CI report: * f89a460408980b0acfcf49854234dada412095e0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=159)