Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 merged PR #11440: URL: https://github.com/apache/hudi/pull/11440 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apac

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

lokeshj1703 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1683797553 ## hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java: ## @@ -365,20 +370,56 @@ public static Option getEarliestInstantToRetainForClustering

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2238212778 ## CI report: * e644864df1edaf92ad2eeb6b77d1a5468528398c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2238310021 ## CI report: * e644864df1edaf92ad2eeb6b77d1a5468528398c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2238522152 ## CI report: * f4c05a461c3603f87ea19fc2e11aca69ad75fad3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1683651944 ## hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java: ## @@ -365,20 +370,56 @@ public static Option getEarliestInstantToRetainForClustering(

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

nsivabalan commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1683633218 ## hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java: ## @@ -365,20 +370,56 @@ public static Option getEarliestInstantToRetainForClustering(

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

nsivabalan commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2237812627 hey @lokeshj1703 : can you update the PR desc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1683622608 ## hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java: ## @@ -365,20 +370,56 @@ public static Option getEarliestInstantToRetainForClustering(

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1683621408 ## hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java: ## @@ -333,28 +335,31 @@ public static List getPendingClusteringInstantTimes(HoodieTabl

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2237186474 ## CI report: * e644864df1edaf92ad2eeb6b77d1a5468528398c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2237118823 ## CI report: * 8769cd80906eb931df92e33211beee8d4d9a7497 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2237106014 ## CI report: * 8769cd80906eb931df92e33211beee8d4d9a7497 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2236822194 ## CI report: * 8769cd80906eb931df92e33211beee8d4d9a7497 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2236688545 ## CI report: * 56188d58f0289b2c6abcf6153bd995746b7f8307 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2236654713 ## CI report: * 56188d58f0289b2c6abcf6153bd995746b7f8307 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2235123582 @lokeshj1703 Thanks for the contribution, I have reviewed and created a patch: [7779.patch.zip](https://github.com/user-attachments/files/16276593/7779.patch.zip) -- This is a

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2234158809 ## CI report: * 56188d58f0289b2c6abcf6153bd995746b7f8307 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2233960417 ## CI report: * 228d9a1568699f5fa9c7372af6159b78c9189256 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2233938345 ## CI report: * 228d9a1568699f5fa9c7372af6159b78c9189256 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

lokeshj1703 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1681475904 ## hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java: ## @@ -376,9 +381,75 @@ public static Option getEarliestInstantToRetainForClustering(

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

lokeshj1703 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1681470626 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java: ## @@ -149,17 +150,24 @@ HoodieCleanerPlan requestCl

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1680216558 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java: ## @@ -149,17 +150,24 @@ HoodieCleanerPlan requestClea

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1680216558 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java: ## @@ -149,17 +150,24 @@ HoodieCleanerPlan requestClea

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2230613582 ## CI report: * 228d9a1568699f5fa9c7372af6159b78c9189256 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2230535145 ## CI report: * 0ccf2a1257697222f488723c7ae3fde9cd7d6b8d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2229174194 ## CI report: * 0ccf2a1257697222f488723c7ae3fde9cd7d6b8d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2229097599 ## CI report: * 15b2428c2dd7d7980fd98b78643811652e08fd35 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

lokeshj1703 commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2229085411 @danny0405 I have updated the PR to address your comments. Can you please check? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1674899742 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java: ## @@ -437,6 +437,15 @@ static boolean compareTimestamps(String commit1, BiPre

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1674895597 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java: ## @@ -149,17 +150,24 @@ HoodieCleanerPlan requestClea

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1674898261 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java: ## @@ -553,13 +552,62 @@ public Pair> getDeletePaths(String part

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1674895597 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java: ## @@ -149,17 +150,24 @@ HoodieCleanerPlan requestClea

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1674893075 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java: ## @@ -264,20 +268,46 @@ private List getCommitInstantsToA

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-704418 ## CI report: * 15b2428c2dd7d7980fd98b78643811652e08fd35 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-595823 ## CI report: * 7e8e5336ab1f079d93ccaa0981e70798169532f1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-511800 ## CI report: * 7e8e5336ab1f079d93ccaa0981e70798169532f1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

lokeshj1703 commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-500128 I have added the optimisation. @danny0405 @nsivabalan PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221755217 > did you guys sync up on that. if not, can we sync up and make forward progress. I thought we have synced up and reached concensus in the last stand-up meeting, no? -- This is

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

nsivabalan commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221624233 @danny0405 @lokeshj1703 : are we aligned on any optimization we need on top of the patch? for eg, we were discussing about replace commit timeline right? did you guys sync up on that. i

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221276113 ## CI report: * 7e8e5336ab1f079d93ccaa0981e70798169532f1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221109430 ## CI report: * 4bb072faa37b2ea398144fbbc24deff966153cfa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221096095 ## CI report: * 4bb072faa37b2ea398144fbbc24deff966153cfa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

lokeshj1703 commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221025516 I have explained the scenario in the comments. PTAL @nbalajee @danny0405 I have also fixed the earliestCommitToNotArchive logic to take into account last clean instant and removed s

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

nbalajee commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2215519646 Let's call out that for replacecommit (clustering operation with replace fileIDs), archiving a replacecommit before the replaced files are cleaned, would result in a data quality issue fo

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2204767885 > Whose value will either refer to first savepoint if first savepoint < earliest commit to retain. Does this mean we can not archive beyond any savepoint now? > Last complet

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

jonvex commented on code in PR #11440: URL: https://github.com/apache/hudi/pull/11440#discussion_r1642831308 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java: ## @@ -362,6 +367,25 @@ private boolean deleteArchivedInstant

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

danny0405 commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2165026195 @nsivabalan Is this the fix we talked before for savepoint deletion? Do we introduce cyclic dependency for clean and archiving. -- This is an automated message from the Apache Git Serv

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2163201030 ## CI report: * 22d7e90f753b9cd7374312c8ff73851453cf9736 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2163084989 ## CI report: * 22d7e90f753b9cd7374312c8ff73851453cf9736 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24

Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]

hudi-bot commented on PR #11440: URL: https://github.com/apache/hudi/pull/11440#issuecomment-2163065510 ## CI report: * 22d7e90f753b9cd7374312c8ff73851453cf9736 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th