[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

2023-08-24 Thread via GitHub
majian1998 commented on code in PR #9472: URL: https://github.com/apache/hudi/pull/9472#discussion_r1305191846 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache

[jira] [Updated] (HUDI-6753) Fix parquet inline reading flaky test

2023-08-24 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-6753: -- Epic Link: HUDI-4302 > Fix parquet inline reading flaky test >

[jira] [Created] (HUDI-6753) Fix parquet inline reading flaky test

2023-08-24 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-6753: - Summary: Fix parquet inline reading flaky test Key: HUDI-6753 URL: https://issues.apache.org/jira/browse/HUDI-6753 Project: Apache Hudi Issue

[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

2023-08-24 Thread via GitHub
majian1998 commented on code in PR #9472: URL: https://github.com/apache/hudi/pull/9472#discussion_r1305179990 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache

[jira] [Comment Edited] (HUDI-1623) Support start_commit_time & end_commit_times for serializable incremental pull

2023-08-24 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758834#comment-17758834 ] Vinoth Chandar edited comment on HUDI-1623 at 8/25/23 5:40 AM: --- On TrueTime,

[GitHub] [hudi] hudi-bot commented on pull request #9531: [HUDI-6445] Triage ci flakiness aug24 2023 attempt7

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9531: URL: https://github.com/apache/hudi/pull/9531#issuecomment-1692790494 ## CI report: * 2b4485a87838bc599b6d5c8c868d4098a34feae0 Azure:

[GitHub] [hudi] tchow-notion commented on issue #9289: [SUPPORT] ConcurrentModificationException in BaseHoodieTableFileIndex.getTotalCachedFilesSize()

2023-08-24 Thread via GitHub
tchow-notion commented on issue #9289: URL: https://github.com/apache/hudi/issues/9289#issuecomment-1692786968 Hi folks, is there an update on this issue? We are experiencing this with concurrent reads. -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Commented] (HUDI-1623) Support start_commit_time & end_commit_times for serializable incremental pull

2023-08-24 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758834#comment-17758834 ] Vinoth Chandar commented on HUDI-1623: -- On TrueTime, we add a new _TrueTimeGenerator_ Interface. By

[jira] [Commented] (HUDI-1623) Support start_commit_time & end_commit_times for serializable incremental pull

2023-08-24 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758833#comment-17758833 ] Vinoth Chandar commented on HUDI-1623: -- On the naming of the active timeline instants:  let's

[jira] [Updated] (HUDI-6708) Support Record Index with the Async Indexer

2023-08-24 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-6708: - Reviewers: Vinoth Chandar > Support Record Index with the Async Indexer >

[GitHub] [hudi] hudi-bot commented on pull request #9533: [HUDI-6445] Fixing metrics in tests

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9533: URL: https://github.com/apache/hudi/pull/9533#issuecomment-1692750092 ## CI report: * 78ba0da62104725400b6aec04775f1e2c7dfd113 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9532: [HUDI-6445] Triage ci flakiness aug24 2023 attempt8

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9532: URL: https://github.com/apache/hudi/pull/9532#issuecomment-1692750065 ## CI report: * c2c0d27633185197448a772bf437424c6c8b212b Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9531: [HUDI-6445] Triage ci flakiness aug24 2023 attempt7

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9531: URL: https://github.com/apache/hudi/pull/9531#issuecomment-1692750049 ## CI report: * 2b4485a87838bc599b6d5c8c868d4098a34feae0 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9530: [HUDI-6445] Triage ci flakiness aug24 2023 attempt6

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9530: URL: https://github.com/apache/hudi/pull/9530#issuecomment-1692750002 ## CI report: * 9c10982d6aefc758e345826ab7a42d5c48371cb5 Azure:

[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-24 Thread via GitHub
danny0405 commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1305108862 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java: ## @@ -424,9 +159,7 @@ private Stream getCommitInstantsToArchive()

[GitHub] [hudi] hudi-bot commented on pull request #9533: [HUDI-6445] Fixing metrics in tests

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9533: URL: https://github.com/apache/hudi/pull/9533#issuecomment-1692722141 ## CI report: * 78ba0da62104725400b6aec04775f1e2c7dfd113 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9532: [HUDI-6445] Triage ci flakiness aug24 2023 attempt8

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9532: URL: https://github.com/apache/hudi/pull/9532#issuecomment-1692722079 ## CI report: * c2c0d27633185197448a772bf437424c6c8b212b Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9531: [HUDI-6445] Triage ci flakiness aug24 2023 attempt7

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9531: URL: https://github.com/apache/hudi/pull/9531#issuecomment-1692722033 ## CI report: * 2b4485a87838bc599b6d5c8c868d4098a34feae0 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9530: [HUDI-6445] Triage ci flakiness aug24 2023 attempt6

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9530: URL: https://github.com/apache/hudi/pull/9530#issuecomment-1692721989 ## CI report: * 9c10982d6aefc758e345826ab7a42d5c48371cb5 Azure:

[jira] [Updated] (HUDI-6722) Performance and API improvement on record merging

2023-08-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6722: Epic Name: Performance and API improvement on record merging > Performance and API improvement on record

[jira] [Updated] (HUDI-6752) Scope out the work for file group reading and writing with record merging in Spark

2023-08-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6752: Status: In Progress (was: Open) > Scope out the work for file group reading and writing with record

[jira] [Created] (HUDI-6752) Scope out the work for file group reading and writing with record merging in Spark

2023-08-24 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-6752: --- Summary: Scope out the work for file group reading and writing with record merging in Spark Key: HUDI-6752 URL: https://issues.apache.org/jira/browse/HUDI-6752 Project: Apache

[jira] [Updated] (HUDI-6752) Scope out the work for file group reading and writing with record merging in Spark

2023-08-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6752: Fix Version/s: 1.0.0 > Scope out the work for file group reading and writing with record merging in >

[jira] [Assigned] (HUDI-6752) Scope out the work for file group reading and writing with record merging in Spark

2023-08-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-6752: --- Assignee: Ethan Guo > Scope out the work for file group reading and writing with record merging in

[jira] [Updated] (HUDI-6751) Scope out remaining work for the record merging API

2023-08-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6751: Status: In Progress (was: Open) > Scope out remaining work for the record merging API >

[jira] [Assigned] (HUDI-6751) Scope out remaining work for the record merging API

2023-08-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-6751: --- Assignee: Ethan Guo > Scope out remaining work for the record merging API >

[jira] [Created] (HUDI-6751) Scope out remaining work for the record merging API

2023-08-24 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-6751: --- Summary: Scope out remaining work for the record merging API Key: HUDI-6751 URL: https://issues.apache.org/jira/browse/HUDI-6751 Project: Apache Hudi Issue Type: Task

[jira] [Updated] (HUDI-6751) Scope out remaining work for the record merging API

2023-08-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6751: Fix Version/s: 1.0.0 > Scope out remaining work for the record merging API >

[jira] [Closed] (HUDI-6721) Prototype and benchmark partial updates in MOR log merging

2023-08-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-6721. --- Resolution: Fixed > Prototype and benchmark partial updates in MOR log merging >

[jira] [Closed] (HUDI-6723) Prototype and benchmark event-time based in MOR log merging

2023-08-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-6723. --- Resolution: Fixed > Prototype and benchmark event-time based in MOR log merging >

[jira] [Closed] (HUDI-6720) Prototype and benchmark position- and key-based updates and deletes in MOR

2023-08-24 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-6720. --- Resolution: Fixed > Prototype and benchmark position- and key-based updates and deletes in MOR >

[GitHub] [hudi] hudi-bot commented on pull request #9532: [HUDI-6445] Triage ci flakiness aug24 2023 attempt8

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9532: URL: https://github.com/apache/hudi/pull/9532#issuecomment-1692716897 ## CI report: * c2c0d27633185197448a772bf437424c6c8b212b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9530: [HUDI-6445] Triage ci flakiness aug24 2023 attempt6

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9530: URL: https://github.com/apache/hudi/pull/9530#issuecomment-1692716804 ## CI report: * 9c10982d6aefc758e345826ab7a42d5c48371cb5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9533: [HUDI-6445] Fixing metrics in tests

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9533: URL: https://github.com/apache/hudi/pull/9533#issuecomment-1692716928 ## CI report: * 78ba0da62104725400b6aec04775f1e2c7dfd113 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9531: [HUDI-6445] Triage ci flakiness aug24 2023 attempt7

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9531: URL: https://github.com/apache/hudi/pull/9531#issuecomment-1692716851 ## CI report: * 2b4485a87838bc599b6d5c8c868d4098a34feae0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9515: [HUDI-2141] Support flink compaction metrics

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9515: URL: https://github.com/apache/hudi/pull/9515#issuecomment-1692716707 ## CI report: * 4ccfb5f43f82a7c57de83794ebaa9c5a051c679a Azure:

[GitHub] [hudi] guanziyue commented on a diff in pull request #9523: [DO NOT MEGRE] HUDI-1517 Adding support for Per log file marker to track all log files added by a commit and to assist with rollbac

2023-08-24 Thread via GitHub
guanziyue commented on code in PR #9523: URL: https://github.com/apache/hudi/pull/9523#discussion_r1305098875 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java: ## @@ -103,35 +120,47 @@ public List

[GitHub] [hudi] nsivabalan commented on pull request #9532: [HUDI-6445] Triage ci flakiness aug24 2023 attempt8

2023-08-24 Thread via GitHub
nsivabalan commented on PR #9532: URL: https://github.com/apache/hudi/pull/9532#issuecomment-1692713081 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] nsivabalan commented on pull request #9531: [HUDI-6445] Triage ci flakiness aug24 2023 attempt7

2023-08-24 Thread via GitHub
nsivabalan commented on PR #9531: URL: https://github.com/apache/hudi/pull/9531#issuecomment-1692713000 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] nsivabalan commented on pull request #9530: [HUDI-6445] Triage ci flakiness aug24 2023 attempt6

2023-08-24 Thread via GitHub
nsivabalan commented on PR #9530: URL: https://github.com/apache/hudi/pull/9530#issuecomment-1692712824 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] stream2000 commented on pull request #9515: [HUDI-2141] Support flink compaction metrics

2023-08-24 Thread via GitHub
stream2000 commented on PR #9515: URL: https://github.com/apache/hudi/pull/9515#issuecomment-1692712087 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] nsivabalan opened a new pull request, #9533: [HUDI-6445] Fixing metrics in tests

2023-08-24 Thread via GitHub
nsivabalan opened a new pull request, #9533: URL: https://github.com/apache/hudi/pull/9533 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any

[GitHub] [hudi] hudi-bot commented on pull request #9528: [HUDI-6445] Triage ci flakiness aug24 2023 attempt5

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9528: URL: https://github.com/apache/hudi/pull/9528#issuecomment-1692711970 ## CI report: * 198827246a0508c418eed0824ea4772f5e14303b Azure:

[GitHub] [hudi] guanziyue commented on a diff in pull request #9523: [DO NOT MEGRE] HUDI-1517 Adding support for Per log file marker to track all log files added by a commit and to assist with rollbac

2023-08-24 Thread via GitHub
guanziyue commented on code in PR #9523: URL: https://github.com/apache/hudi/pull/9523#discussion_r1305094068 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/marker/DirectWriteMarkers.java: ## @@ -130,6 +132,53 @@ public Set

[GitHub] [hudi] nsivabalan closed pull request #9527: [HUDI-6445] Triage ci flakiness aug24 2023 attempt4

2023-08-24 Thread via GitHub
nsivabalan closed pull request #9527: [HUDI-6445] Triage ci flakiness aug24 2023 attempt4 URL: https://github.com/apache/hudi/pull/9527 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] nsivabalan closed pull request #9526: [HUDI-6445] Triage ci flakiness aug24 2023 attempt3

2023-08-24 Thread via GitHub
nsivabalan closed pull request #9526: [HUDI-6445] Triage ci flakiness aug24 2023 attempt3 URL: https://github.com/apache/hudi/pull/9526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] nsivabalan opened a new pull request, #9532: [HUDI-6445] Triage ci flakiness aug24 2023 attempt8

2023-08-24 Thread via GitHub
nsivabalan opened a new pull request, #9532: URL: https://github.com/apache/hudi/pull/9532 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any

[GitHub] [hudi] nsivabalan opened a new pull request, #9531: [HUDI-6445] Triage ci flakiness aug24 2023 attempt7

2023-08-24 Thread via GitHub
nsivabalan opened a new pull request, #9531: URL: https://github.com/apache/hudi/pull/9531 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any

[GitHub] [hudi] nsivabalan opened a new pull request, #9530: [HUDI-6445] Triage ci flakiness aug24 2023 attempt6

2023-08-24 Thread via GitHub
nsivabalan opened a new pull request, #9530: URL: https://github.com/apache/hudi/pull/9530 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any

[jira] [Updated] (HUDI-6710) Initial investigation to unearth gaps and unearth more open items.

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6710: -- Status: Patch Available (was: In Progress) > Initial investigation to unearth gaps and unearth more

[jira] [Commented] (HUDI-6710) Initial investigation to unearth gaps and unearth more open items.

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758801#comment-17758801 ] Sagar Sumit commented on HUDI-6710: --- I think we are clear what we need to do to support multi-table

[jira] [Closed] (HUDI-6710) Initial investigation to unearth gaps and unearth more open items.

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-6710. - Resolution: Done > Initial investigation to unearth gaps and unearth more open items. >

[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-24 Thread via GitHub
danny0405 commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1305083315 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveAction.java: ## @@ -0,0 +1,162 @@ +/* + * Licensed to the Apache Software Foundation

[jira] [Updated] (HUDI-6749) Support BEGIN...END SQL syntax for multi-table transaction

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6749: -- Fix Version/s: 1.0.0 > Support BEGIN...END SQL syntax for multi-table transaction >

[jira] [Updated] (HUDI-6750) Enhance ConflictResolutionStrategy to resolve conflicts at database level

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6750: -- Fix Version/s: 1.0.0 > Enhance ConflictResolutionStrategy to resolve conflicts at database level >

[jira] [Updated] (HUDI-6748) Enhance TransactionManager to support multi-table transaction

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6748: -- Fix Version/s: 1.0.0 > Enhance TransactionManager to support multi-table transaction >

[jira] [Updated] (HUDI-6748) Enhance TransactionManager to support multi-table transaction

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6748: -- Description: Transaction Manager should be database timeline-aware and return a transaction instance

[jira] [Updated] (HUDI-6745) Make Spark HoodieCatalog transaction-aware

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6745: -- Fix Version/s: 1.0.0 > Make Spark HoodieCatalog transaction-aware >

[jira] [Updated] (HUDI-6746) Add database timeline and new actions to support multi-table transactions

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6746: -- Fix Version/s: 1.0.0 > Add database timeline and new actions to support multi-table transactions >

[jira] [Updated] (HUDI-6744) Support notion of database in HoodieCatalog

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6744: -- Fix Version/s: 1.0.0 > Support notion of database in HoodieCatalog >

[jira] [Assigned] (HUDI-6744) Support notion of database in HoodieCatalog

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-6744: - Assignee: Sagar Sumit > Support notion of database in HoodieCatalog >

[GitHub] [hudi] voonhous commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

2023-08-24 Thread via GitHub
voonhous commented on code in PR #9472: URL: https://github.com/apache/hudi/pull/9472#discussion_r1305077926 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache

[jira] [Assigned] (HUDI-6749) Support BEGIN...END SQL syntax for multi-table transaction

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-6749: - Assignee: Sagar Sumit > Support BEGIN...END SQL syntax for multi-table transaction >

[jira] [Assigned] (HUDI-6748) Enhance TransactionManager to support multi-table transaction

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-6748: - Assignee: Sagar Sumit > Enhance TransactionManager to support multi-table transaction >

[jira] [Assigned] (HUDI-6745) Make Spark HoodieCatalog transaction-aware

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-6745: - Assignee: Sagar Sumit > Make Spark HoodieCatalog transaction-aware >

[jira] [Assigned] (HUDI-6750) Enhance ConflictResolutionStrategy to resolve conflicts at database level

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-6750: - Assignee: Sagar Sumit > Enhance ConflictResolutionStrategy to resolve conflicts at database

[jira] [Assigned] (HUDI-6746) Add database timeline and new actions to support multi-table transactions

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-6746: - Assignee: Sagar Sumit > Add database timeline and new actions to support multi-table

[jira] [Created] (HUDI-6750) Enhance ConflictResolutionStrategy to resolve conflicts at database level

2023-08-24 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-6750: - Summary: Enhance ConflictResolutionStrategy to resolve conflicts at database level Key: HUDI-6750 URL: https://issues.apache.org/jira/browse/HUDI-6750 Project: Apache Hudi

[jira] [Created] (HUDI-6749) Support BEGIN...END SQL syntax for multi-table transaction

2023-08-24 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-6749: - Summary: Support BEGIN...END SQL syntax for multi-table transaction Key: HUDI-6749 URL: https://issues.apache.org/jira/browse/HUDI-6749 Project: Apache Hudi Issue

[GitHub] [hudi] hudi-bot commented on pull request #9525: [HUDI-6445] Triage ci flakiness aug24 2023 attempt2

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9525: URL: https://github.com/apache/hudi/pull/9525#issuecomment-1692685689 ## CI report: * 2a619f68600e8bd311102ecd8954698e17c6843d Azure:

[jira] [Created] (HUDI-6748) Enhance TransactionManager to support multi-table transaction

2023-08-24 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-6748: - Summary: Enhance TransactionManager to support multi-table transaction Key: HUDI-6748 URL: https://issues.apache.org/jira/browse/HUDI-6748 Project: Apache Hudi

[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-24 Thread via GitHub
danny0405 commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1305073510 ## hudi-common/src/main/avro/HoodieLSMTimelineInstant.avsc: ## @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[jira] [Created] (HUDI-6747) Support decode UDF for plan and commit metadata from the archived timeline

2023-08-24 Thread Danny Chen (Jira)
Danny Chen created HUDI-6747: Summary: Support decode UDF for plan and commit metadata from the archived timeline Key: HUDI-6747 URL: https://issues.apache.org/jira/browse/HUDI-6747 Project: Apache Hudi

[jira] [Created] (HUDI-6746) Add database timeline and new actions to support multi-table transactions

2023-08-24 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-6746: - Summary: Add database timeline and new actions to support multi-table transactions Key: HUDI-6746 URL: https://issues.apache.org/jira/browse/HUDI-6746 Project: Apache Hudi

[jira] [Updated] (HUDI-6744) Support notion of database in HoodieCatalog

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6744: -- Summary: Support notion of database in HoodieCatalog (was: Support database and database timeline in

[jira] [Updated] (HUDI-6745) Make Spark HoodieCatalog transaction-aware

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6745: -- Summary: Make Spark HoodieCatalog transaction-aware (was: Make HoodieCatalog transaction-aware) >

[GitHub] [hudi] voonhous commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

2023-08-24 Thread via GitHub
voonhous commented on code in PR #9472: URL: https://github.com/apache/hudi/pull/9472#discussion_r1305067820 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache

[jira] [Created] (HUDI-6745) Make HoodieCatalog transaction-aware

2023-08-24 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-6745: - Summary: Make HoodieCatalog transaction-aware Key: HUDI-6745 URL: https://issues.apache.org/jira/browse/HUDI-6745 Project: Apache Hudi Issue Type: Task

[jira] [Created] (HUDI-6744) Support database and database timeline in HoodieCatalog

2023-08-24 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-6744: - Summary: Support database and database timeline in HoodieCatalog Key: HUDI-6744 URL: https://issues.apache.org/jira/browse/HUDI-6744 Project: Apache Hudi Issue

[jira] [Updated] (HUDI-6711) Write a RFC for Multi Table Txns

2023-08-24 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6711: -- Status: In Progress (was: Open) > Write a RFC for Multi Table Txns >

[GitHub] [hudi] hudi-bot commented on pull request #9523: [DO NOT MEGRE] HUDI-1517-new with nsivabalan

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9523: URL: https://github.com/apache/hudi/pull/9523#issuecomment-1692645105 ## CI report: * f04d3fb5e5400c0dc6e4a091e7251381fa00a743 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9521: [HUDI-6736] Revert pr 8849

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9521: URL: https://github.com/apache/hudi/pull/9521#issuecomment-1692639315 ## CI report: * 0aa97d414fd91d95e5931d108407dbc2b280b519 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9526: [HUDI-6445] Triage ci flakiness aug24 2023 attempt3

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9526: URL: https://github.com/apache/hudi/pull/9526#issuecomment-1692631752 ## CI report: * 418646378e3c1a7a949188c6ea9ed74629a7f8d4 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9527: [HUDI-6445] Triage ci flakiness aug24 2023 attempt4

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9527: URL: https://github.com/apache/hudi/pull/9527#issuecomment-1692631780 ## CI report: * 418646378e3c1a7a949188c6ea9ed74629a7f8d4 Azure:

[jira] [Comment Edited] (HUDI-6712) Implement optimized keyed lookup on parquet files

2023-08-24 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758787#comment-17758787 ] Lin Liu edited comment on HUDI-6712 at 8/25/23 1:33 AM: During our process of

[jira] [Comment Edited] (HUDI-6712) Implement optimized keyed lookup on parquet files

2023-08-24 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758787#comment-17758787 ] Lin Liu edited comment on HUDI-6712 at 8/25/23 1:31 AM: During our process of

[jira] [Comment Edited] (HUDI-6712) Implement optimized keyed lookup on parquet files

2023-08-24 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758787#comment-17758787 ] Lin Liu edited comment on HUDI-6712 at 8/25/23 1:31 AM: During our process of

[jira] [Commented] (HUDI-6712) Implement optimized keyed lookup on parquet files

2023-08-24 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758787#comment-17758787 ] Lin Liu commented on HUDI-6712: --- During our process of moving lake_plumber code into hudi, we found that the

[GitHub] [hudi] njalan opened a new issue, #9529: S3 storage cpu is high when spark streaming is upserting data into hudi

2023-08-24 Thread via GitHub
njalan opened a new issue, #9529: URL: https://github.com/apache/hudi/issues/9529 S3 storage cpu is high when spark streaming is upserting data into hudi. it is local storage works like s3 and when the spark steaming jobs is more the cpu is pretty high but I/O is not high. A

[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-24 Thread via GitHub
vinothchandar commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1300866432 ## hudi-common/src/main/avro/HoodieLSMTimelineInstant.avsc: ## @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[GitHub] [hudi] hudi-bot commented on pull request #9527: [HUDI-6445] Triage ci flakiness aug24 2023 attempt4

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9527: URL: https://github.com/apache/hudi/pull/9527#issuecomment-1692596305 ## CI report: * 418646378e3c1a7a949188c6ea9ed74629a7f8d4 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9526: [HUDI-6445] Triage ci flakiness aug24 2023 attempt3

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9526: URL: https://github.com/apache/hudi/pull/9526#issuecomment-1692596296 ## CI report: * 418646378e3c1a7a949188c6ea9ed74629a7f8d4 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9522: [HUDI-6445] Triaging flaky tests attempt 1

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9522: URL: https://github.com/apache/hudi/pull/9522#issuecomment-1692596267 ## CI report: * 958e9c9754ecea90afeea4f59ea56b6182954975 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9527: [HUDI-6445] Triage ci flakiness aug24 2023 attempt4

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9527: URL: https://github.com/apache/hudi/pull/9527#issuecomment-1692591201 ## CI report: * 418646378e3c1a7a949188c6ea9ed74629a7f8d4 Azure:

[GitHub] [hudi] danny0405 commented on pull request #9477: [HUDI-6726] Fix connection leaks related to file reader and iterator close

2023-08-24 Thread via GitHub
danny0405 commented on PR #9477: URL: https://github.com/apache/hudi/pull/9477#issuecomment-1692590458 Is this a blocker for release 0.14.0? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9523: [DO NOT MEGRE] HUDI-1517-new with nsivabalan

2023-08-24 Thread via GitHub
nsivabalan commented on code in PR #9523: URL: https://github.com/apache/hudi/pull/9523#discussion_r1304953163 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/marker/DirectWriteMarkers.java: ## @@ -130,6 +132,53 @@ public Set

[GitHub] [hudi] hudi-bot commented on pull request #9527: [HUDI-6445] Triage ci flakiness aug24 2023 attempt4

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9527: URL: https://github.com/apache/hudi/pull/9527#issuecomment-1692559152 ## CI report: * 418646378e3c1a7a949188c6ea9ed74629a7f8d4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9524: [MINOR][DNM] Release 0.12.2 patch

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9524: URL: https://github.com/apache/hudi/pull/9524#issuecomment-1692559109 ## CI report: * ed246912f19b690a253b25a2b9bcc3a380a798b2 UNKNOWN * 8eec67f01a8b06c4608fef250b165622dcbc0b99 UNKNOWN Bot commands @hudi-bot supports the

[GitHub] [hudi] Riddle4045 commented on issue #9495: [SUPPORT] Writing Hudi tables with Flink fails with HFile exceptions

2023-08-24 Thread via GitHub
Riddle4045 commented on issue #9495: URL: https://github.com/apache/hudi/issues/9495#issuecomment-1692520867 @danny0405 I repackaged from branch `release-0.13.0` using ` mvn clean package -DskipTests -Dspark3.4 -Dflink1.16 -Pflink-bundle-shade-hive3` but no luck, any pointers on how I

[GitHub] [hudi] hudi-bot commented on pull request #9528: [HUDI-6445] Triage ci flakiness aug24 2023 attempt5

2023-08-24 Thread via GitHub
hudi-bot commented on PR #9528: URL: https://github.com/apache/hudi/pull/9528#issuecomment-1692519922 ## CI report: * 198827246a0508c418eed0824ea4772f5e14303b Azure:

  1   2   >