[I] [SUPPORT] Duplicate fileId exception occurs in Flink Bucket MOR table [hudi]

2024-08-15 Thread via GitHub
usberkeley opened a new issue, #11784: URL: https://github.com/apache/hudi/issues/11784 **Describe the problem you faced** When Flink is restarted, a Duplicate fileId exception occurs in the Flink Bucket MOR table **To Reproduce** Steps to reproduce the behavior:

Re: [PR] [HUDI-7989] Fix secondary index updates [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11634: URL: https://github.com/apache/hudi/pull/11634#issuecomment-2292872680 ## CI report: * b635082f240f6238c6908362a63570f92c71b07f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=71)

Re: [PR] [HUDI-7989] Fix secondary index updates [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11634: URL: https://github.com/apache/hudi/pull/11634#issuecomment-2292830613 ## CI report: * b635082f240f6238c6908362a63570f92c71b07f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=71)

Re: [PR] [HUDI-7989] Fix secondary index updates [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11634: URL: https://github.com/apache/hudi/pull/11634#issuecomment-2292829674 ## CI report: * 969884fcb6e3de032bd656c8e85823ced1a999fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25

Re: [PR] [HUDI-7989] Fix secondary index updates [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11634: URL: https://github.com/apache/hudi/pull/11634#issuecomment-2292812030 ## CI report: * 969884fcb6e3de032bd656c8e85823ced1a999fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25

Re: [PR] [HUDI-7989] Fix secondary index updates [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11634: URL: https://github.com/apache/hudi/pull/11634#issuecomment-229289 ## CI report: * 969884fcb6e3de032bd656c8e85823ced1a999fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25

Re: [PR] [HUDI-8087] don't start docker in the build phase in integration tests [hudi]

2024-08-15 Thread via GitHub
jonvex closed pull request #11782: [HUDI-8087] don't start docker in the build phase in integration tests URL: https://github.com/apache/hudi/pull/11782 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
jonvex merged PR #11692: URL: https://github.com/apache/hudi/pull/11692 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292678055 ## CI report: * 773452ef30d42be1425f7886cd4de331333f0ec7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=69)

Re: [PR] [HUDI-8087] don't start docker in the build phase in integration tests [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292652309 ## CI report: * f1d896889ed5076654b7cb2c47890d6b05fda889 UNKNOWN * fee76a4a3829724d61fbd115c15bef5b82e8003a UNKNOWN * ea228be1aea8acb74d8d31151324daa114381a0e Azure: [SUCC

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292621726 ## CI report: * 161f011fba8d60775cf47a295732d43a4fcba7db Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=67)

Re: [PR] [HUDI-8087] don't start docker in the build phase in integration tests [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292598177 ## CI report: * f1d896889ed5076654b7cb2c47890d6b05fda889 UNKNOWN * fee76a4a3829724d61fbd115c15bef5b82e8003a UNKNOWN * c773a5c339579c545dce55876833b3bb125258f0 Azure: [SUCC

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292574463 ## CI report: * 161f011fba8d60775cf47a295732d43a4fcba7db Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=67)

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292572839 ## CI report: * 216e2dc40077a19bca6d6b7ee36125a976a55607 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=65) *

Re: [PR] [HUDI-8087] don't start docker in the build phase in integration tests [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292570802 ## CI report: * f1d896889ed5076654b7cb2c47890d6b05fda889 UNKNOWN * fee76a4a3829724d61fbd115c15bef5b82e8003a UNKNOWN * c773a5c339579c545dce55876833b3bb125258f0 Azure: [SUCC

[jira] [Updated] (HUDI-8087) Prevent docker from starting in the build phase of integration tests

2024-08-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-8087: - Labels: pull-request-available (was: ) > Prevent docker from starting in the build phase of integ

[jira] [Closed] (HUDI-8067) Docker Compose V1 removed from 6.5.0-1025-azure

2024-08-15 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler closed HUDI-8067. - Resolution: Fixed > Docker Compose V1 removed from 6.5.0-1025-azure >

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292558507 ## CI report: * 216e2dc40077a19bca6d6b7ee36125a976a55607 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=65) *

[jira] [Created] (HUDI-8087) Prevent docker from starting in the build phase of integration tests

2024-08-15 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-8087: - Summary: Prevent docker from starting in the build phase of integration tests Key: HUDI-8087 URL: https://issues.apache.org/jira/browse/HUDI-8087 Project: Apache Hu

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292556014 ## CI report: * 216e2dc40077a19bca6d6b7ee36125a976a55607 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=65) *

Re: [PR] [HUDI-8070] Support Flink 1.19 [hudi]

2024-08-15 Thread via GitHub
danny0405 commented on code in PR #11779: URL: https://github.com/apache/hudi/pull/11779#discussion_r1719159582 ## pom.xml: ## @@ -2721,6 +2722,19 @@ + + flink1.19 + +1.5.6 +1.11.1 +1.13.1 + + + +

Re: [PR] [HUDI-8070] Support Flink 1.19 [hudi]

2024-08-15 Thread via GitHub
danny0405 commented on code in PR #11779: URL: https://github.com/apache/hudi/pull/11779#discussion_r1719158683 ## .github/workflows/bot.yml: ## @@ -570,11 +575,11 @@ jobs: matrix: include: - scalaProfile: 'scala-2.13' -flinkProfile: 'flink

[jira] [Assigned] (HUDI-7843) Support record merge mode with partial updates

2024-08-15 Thread Vova Kolmakov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vova Kolmakov reassigned HUDI-7843: --- Assignee: Vova Kolmakov > Support record merge mode with partial updates > --

[jira] [Updated] (HUDI-8075) Revisit table service scheduling and execution with the completion time

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8075: Status: In Progress (was: Open) > Revisit table service scheduling and execution with the completion time >

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292492361 ## CI report: * f1d896889ed5076654b7cb2c47890d6b05fda889 UNKNOWN * fee76a4a3829724d61fbd115c15bef5b82e8003a UNKNOWN * c773a5c339579c545dce55876833b3bb125258f0 Azure: [SUCC

[jira] [Updated] (HUDI-8036) Handle partition schema for custom key gen in SparkHoodieTableFileIndex

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8036: Sprint: Hudi 1.0 Sprint 2024/08/12-18 > Handle partition schema for custom key gen in SparkHoodieTableFileIn

[jira] [Updated] (HUDI-7902) Partition fields in Table config should store partition field types for custom key generator

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7902: Sprint: (was: Hudi 1.0 Sprint 2024/08/12-18) > Partition fields in Table config should store partition fie

[jira] [Updated] (HUDI-8044) Store timestamp keygen configs for custom key generator

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8044: Epic Link: HUDI-7856 > Store timestamp keygen configs for custom key generator > ---

[jira] [Updated] (HUDI-8044) Store timestamp keygen configs for custom key generator

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8044: Parent: (was: HUDI-7902) Issue Type: Improvement (was: Sub-task) > Store timestamp keygen confi

[jira] [Updated] (HUDI-8036) Handle partition schema for custom key gen in SparkHoodieTableFileIndex

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8036: Parent: (was: HUDI-7902) Issue Type: New Feature (was: Sub-task) > Handle partition schema for

[jira] [Updated] (HUDI-7996) Store partition type with partition fields in table configs

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7996: Epic Link: HUDI-7856 > Store partition type with partition fields in table configs > ---

[jira] [Updated] (HUDI-8036) Handle partition schema for custom key gen in SparkHoodieTableFileIndex

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8036: Epic Link: HUDI-7856 > Handle partition schema for custom key gen in SparkHoodieTableFileIndex > ---

[jira] [Updated] (HUDI-7996) Store partition type with partition fields in table configs

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7996: Parent: (was: HUDI-7902) Issue Type: New Feature (was: Sub-task) > Store partition type with pa

[jira] [Updated] (HUDI-7970) Move methods from HoodieTableConfigUtils to HoodieTableConfig

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7970: Epic Link: HUDI-7856 > Move methods from HoodieTableConfigUtils to HoodieTableConfig > -

[jira] [Updated] (HUDI-7970) Move methods from HoodieTableConfigUtils to HoodieTableConfig

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7970: Parent: (was: HUDI-7902) Issue Type: Improvement (was: Sub-task) > Move methods from HoodieTabl

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292468807 ## CI report: * 216e2dc40077a19bca6d6b7ee36125a976a55607 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=65)

[jira] [Updated] (HUDI-7989) Fix secondary index updates with other indexes

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7989: Sprint: Hudi 1.0 Sprint 2024/08/12-18 > Fix secondary index updates with other indexes > ---

[jira] [Updated] (HUDI-7990) Remove wasteful index dirs under metadata table base path for secondary and functional index

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7990: Sprint: Hudi 1.0 Sprint 2024/08/12-18 > Remove wasteful index dirs under metadata table base path for second

[jira] [Updated] (HUDI-7982) [Umbrella] Issues found with 1.0.0-beta2 multi-modal indexing

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7982: Sprint: (was: Hudi 1.0 Sprint 2024/08/12-18) > [Umbrella] Issues found with 1.0.0-beta2 multi-modal indexi

[jira] [Updated] (HUDI-7994) Support secondary index on nested fields

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7994: Parent: (was: HUDI-7982) Issue Type: Bug (was: Sub-task) > Support secondary index on nested fi

[jira] [Updated] (HUDI-8013) Test Plan for multi-modal index

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8013: Parent: (was: HUDI-7982) Issue Type: Bug (was: Sub-task) > Test Plan for multi-modal index > --

[jira] [Updated] (HUDI-7994) Support secondary index on nested fields

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7994: Epic Link: HUDI-3907 > Support secondary index on nested fields > >

[jira] [Updated] (HUDI-8013) Test Plan for multi-modal index

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8013: Epic Link: HUDI-3907 > Test Plan for multi-modal index > --- > >

[jira] [Updated] (HUDI-7991) Ensure secondary index readable using the native hfile reader

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7991: Parent: (was: HUDI-7982) Issue Type: Bug (was: Sub-task) > Ensure secondary index readable usin

[jira] [Updated] (HUDI-7992) Index updates should not fail with schema evolution

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7992: Epic Link: HUDI-3907 > Index updates should not fail with schema evolution > ---

[jira] [Updated] (HUDI-7991) Ensure secondary index readable using the native hfile reader

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7991: Epic Link: HUDI-3907 > Ensure secondary index readable using the native hfile reader > -

[jira] [Updated] (HUDI-7993) Support pruning and skipping with meta fields

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7993: Parent: (was: HUDI-7982) Issue Type: Bug (was: Sub-task) > Support pruning and skipping with me

[jira] [Updated] (HUDI-7993) Support pruning and skipping with meta fields

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7993: Epic Link: HUDI-3907 > Support pruning and skipping with meta fields > -

[jira] [Updated] (HUDI-7992) Index updates should not fail with schema evolution

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7992: Parent: (was: HUDI-7982) Issue Type: Bug (was: Sub-task) > Index updates should not fail with s

[jira] [Updated] (HUDI-7990) Remove wasteful index dirs under metadata table base path for secondary and functional index

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7990: Epic Link: HUDI-3907 > Remove wasteful index dirs under metadata table base path for secondary and > functi

[jira] [Updated] (HUDI-7990) Remove wasteful index dirs under metadata table base path for secondary and functional index

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7990: Parent: (was: HUDI-7982) Issue Type: Bug (was: Sub-task) > Remove wasteful index dirs under met

[jira] [Updated] (HUDI-7989) Fix secondary index updates with other indexes

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7989: Epic Link: HUDI-3907 > Fix secondary index updates with other indexes >

[jira] [Updated] (HUDI-7989) Fix secondary index updates with other indexes

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7989: Parent: (was: HUDI-7982) Issue Type: Bug (was: Sub-task) > Fix secondary index updates with oth

Re: [I] [SUPPORT] Hudi write to COW table hangs on Preparing compaction metadata job [hudi]

2024-08-15 Thread via GitHub
Gatsby-Lee commented on issue #11712: URL: https://github.com/apache/hudi/issues/11712#issuecomment-2292446150 @ad1happy2go Hi, I don't know much about the log files and the file groups. Why do the size and the number of log files and file groups matter for this issue? -- This is an au

[jira] [Updated] (HUDI-8003) Add overwrite payload for hive for record reader

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8003: Reviewers: Ethan Guo > Add overwrite payload for hive for record reader > --

[jira] [Updated] (HUDI-7919) Make integration tests run on Spark 3.5

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7919: Reviewers: Ethan Guo > Make integration tests run on Spark 3.5 > --- > >

[jira] [Updated] (HUDI-5807) HoodieSparkParquetReader is not appending partition-path values

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5807: Reviewers: Ethan Guo > HoodieSparkParquetReader is not appending partition-path values > ---

[jira] [Updated] (HUDI-8073) Add hosts to storage info and pass from hive reader

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8073: Reviewers: Ethan Guo > Add hosts to storage info and pass from hive reader > ---

[jira] [Updated] (HUDI-8080) Get rid of separate reader instance for cdc reader

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8080: Reviewers: Ethan Guo > Get rid of separate reader instance for cdc reader >

[I] [SUPPORT] Failed to use Bloom filter Indexing [hudi]

2024-08-15 Thread via GitHub
Gatsby-Lee opened a new issue, #11783: URL: https://github.com/apache/hudi/issues/11783 **Describe the problem you faced** * Hudi 0.14.1 * Enabled Metadata Table + Enabled Bloom filter Indexing * When enabling "hoodie.bloom.index.use.metadata=true" to use the Bloom filter Indexi

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292427800 ## CI report: * f1d896889ed5076654b7cb2c47890d6b05fda889 UNKNOWN * 8013377c381ad5afada2074cae5941b6acb29600 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-4

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292426634 ## CI report: * f1d896889ed5076654b7cb2c47890d6b05fda889 UNKNOWN * 8013377c381ad5afada2074cae5941b6acb29600 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-4

[jira] [Updated] (HUDI-8079) Get rid of base file reader usage for count in filegroup reader parquet file format

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8079: Reviewers: Ethan Guo > Get rid of base file reader usage for count in filegroup reader parquet file > forma

[jira] [Updated] (HUDI-5807) HoodieSparkParquetReader is not appending partition-path values

2024-08-15 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5807: -- Status: Patch Available (was: In Progress) > HoodieSparkParquetReader is not appending partitio

[jira] [Updated] (HUDI-5807) HoodieSparkParquetReader is not appending partition-path values

2024-08-15 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5807: -- Status: In Progress (was: Open) > HoodieSparkParquetReader is not appending partition-path valu

[jira] [Updated] (HUDI-5807) HoodieSparkParquetReader is not appending partition-path values

2024-08-15 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5807: -- Status: Open (was: Patch Available) > HoodieSparkParquetReader is not appending partition-path

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292409316 ## CI report: * a8f0a7c79546e79eabaec698411ced19165bb2aa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=58) *

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292408247 ## CI report: * a8f0a7c79546e79eabaec698411ced19165bb2aa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=58) *

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292374308 ## CI report: * f1d896889ed5076654b7cb2c47890d6b05fda889 UNKNOWN * 8013377c381ad5afada2074cae5941b6acb29600 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-4

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292371499 ## CI report: * 6be8790dc5b11988b36aa8b6cc5278155f0f5e89 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=63)

Re: [PR] [HUDI-8073] Add hosts to storage path info and use it if present [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11761: URL: https://github.com/apache/hudi/pull/11761#issuecomment-2292367812 ## CI report: * d82212f838a1482e590537261cb3ab21d920e72f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=62)

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292349014 ## CI report: * 6be8790dc5b11988b36aa8b6cc5278155f0f5e89 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=63)

[jira] [Updated] (HUDI-8079) Get rid of base file reader usage for count in filegroup reader parquet file format

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8079: Story Points: 1 > Get rid of base file reader usage for count in filegroup reader parquet file > format > -

[jira] [Updated] (HUDI-8003) Add overwrite payload for hive for record reader

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8003: Story Points: 12 > Add overwrite payload for hive for record reader > --

[jira] [Updated] (HUDI-8080) Get rid of separate reader instance for cdc reader

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8080: Story Points: 0.5 > Get rid of separate reader instance for cdc reader > ---

[jira] [Updated] (HUDI-8073) Add hosts to storage info and pass from hive reader

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8073: Story Points: 1 (was: 2) > Add hosts to storage info and pass from hive reader > --

[jira] [Updated] (HUDI-8073) Add hosts to storage info and pass from hive reader

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-8073: Story Points: 2 > Add hosts to storage info and pass from hive reader >

[jira] [Updated] (HUDI-7918) Remove support of Spark 2, 3.0, 3.1, and 3.2

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7918: Story Points: 10 (was: 4) > Remove support of Spark 2, 3.0, 3.1, and 3.2 >

[jira] [Updated] (HUDI-7918) Remove support of Spark 2, 3.0, 3.1, and 3.2

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7918: Story Points: 6 (was: 10) > Remove support of Spark 2, 3.0, 3.1, and 3.2 >

[jira] [Closed] (HUDI-7920) Make Spark 3.5 the default build profile for Spark integration

2024-08-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-7920. --- Resolution: Fixed > Make Spark 3.5 the default build profile for Spark integration > -

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292340292 ## CI report: * 6be8790dc5b11988b36aa8b6cc5278155f0f5e89 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=63)

Re: [PR] [HUDI-8066] cherry pick Flink 1.18 into 0.14.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11780: URL: https://github.com/apache/hudi/pull/11780#issuecomment-2292316398 ## CI report: * 3f8d31ac35b892058f1ea818c966f6f056d8225e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=55)

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292307825 ## CI report: * 6be8790dc5b11988b36aa8b6cc5278155f0f5e89 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=63)

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292305383 ## CI report: * 6be8790dc5b11988b36aa8b6cc5278155f0f5e89 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=63) *

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292271676 ## CI report: * a8f0a7c79546e79eabaec698411ced19165bb2aa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=58)

[jira] [Updated] (HUDI-7971) Test and Certify 0.14.x tables are readable in 1.x Hudi reader

2024-08-15 Thread Surya Prasanna Yalla (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surya Prasanna Yalla updated HUDI-7971: --- Summary: Test and Certify 0.14.x tables are readable in 1.x Hudi reader (was: Test a

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292231527 ## CI report: * 6be8790dc5b11988b36aa8b6cc5278155f0f5e89 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=63)

Re: [PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11782: URL: https://github.com/apache/hudi/pull/11782#issuecomment-2292228859 ## CI report: * 6be8790dc5b11988b36aa8b6cc5278155f0f5e89 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[PR] [DO NOT MERGE] don't run it in build phase [hudi]

2024-08-15 Thread via GitHub
jonvex opened a new pull request, #11782: URL: https://github.com/apache/hudi/pull/11782 ### Change Logs asdfdsfsdaf ### Impact sdgfsadsafsadf ### Risk level (write none, low medium or high below) none ### Documentation Update N/A ### Contri

Re: [PR] [HUDI-8073] Add hosts to storage path info and use it if present [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11761: URL: https://github.com/apache/hudi/pull/11761#issuecomment-2292161097 ## CI report: * 629f63c7dcf50673e132e6ebdd4f49d96dc58b44 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=46) *

Re: [PR] [HUDI-8073] Add hosts to storage path info and use it if present [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11761: URL: https://github.com/apache/hudi/pull/11761#issuecomment-2292157864 ## CI report: * 629f63c7dcf50673e132e6ebdd4f49d96dc58b44 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=46) *

[jira] [Closed] (HUDI-8080) Get rid of separate reader instance for cdc reader

2024-08-15 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler closed HUDI-8080. - Resolution: Fixed > Get rid of separate reader instance for cdc reader > -

[jira] [Closed] (HUDI-8079) Get rid of base file reader usage for count in filegroup reader parquet file format

2024-08-15 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler closed HUDI-8079. - Resolution: Fixed > Get rid of base file reader usage for count in filegroup reader parquet file

(hudi) branch master updated: use parquet reader for cdc reading (#11775)

2024-08-15 Thread jonvex
This is an automated email from the ASF dual-hosted git repository. jonvex pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new d4a4d9c43e4 use parquet reader for cdc reading (#1

Re: [PR] [HUDI-8080] use parquet reader for cdc reading instead of creating a separate instance [hudi]

2024-08-15 Thread via GitHub
jonvex merged PR #11775: URL: https://github.com/apache/hudi/pull/11775 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292094186 ## CI report: * a8f0a7c79546e79eabaec698411ced19165bb2aa Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=58)

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292090369 ## CI report: * a8f0a7c79546e79eabaec698411ced19165bb2aa UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
jonvex commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292090863 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [HUDI-7918] Remove support of Spark 3.0, 3.1, and 3.2 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11692: URL: https://github.com/apache/hudi/pull/11692#issuecomment-2292055481 ## CI report: * a8f0a7c79546e79eabaec698411ced19165bb2aa UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-8070] Support Flink 1.19 [hudi]

2024-08-15 Thread via GitHub
hudi-bot commented on PR #11779: URL: https://github.com/apache/hudi/pull/11779#issuecomment-2292034271 ## CI report: * 7984ae9e7e580f821a094320238f6262ac5772b2 UNKNOWN * 02063e180d4d6f8158a400b4afbe0bca62b49234 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47

  1   2   >