amogh-jahagirdar merged PR #10983:
URL: https://github.com/apache/iceberg/pull/10983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@
hantangwangd commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1739706862
##
core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java:
##
@@ -370,7 +370,7 @@ public void testRetainLastWithExpireById() {
}
// Retain la
amogh-jahagirdar commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1739152717
##
core/src/main/java/org/apache/iceberg/RemoveSnapshots.java:
##
@@ -321,6 +323,15 @@ ExpireSnapshots withIncrementalCleanup(boolean
useIncrementalCleanup)
hantangwangd commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1737646125
##
core/src/main/java/org/apache/iceberg/RemoveSnapshots.java:
##
@@ -116,6 +117,7 @@ public ExpireSnapshots cleanExpiredFiles(boolean clean) {
public ExpireS
amogh-jahagirdar commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1737259547
##
core/src/main/java/org/apache/iceberg/RemoveSnapshots.java:
##
@@ -321,6 +323,14 @@ ExpireSnapshots withIncrementalCleanup(boolean
useIncrementalCleanup)
hantangwangd commented on PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#issuecomment-2317476054
Following the discussions above, I changed the approach to just ensure that
users will not fall into a state where they cleanup files that are still
referenced:
- If `expir
hantangwangd commented on PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#issuecomment-2316827563
> The incremental cleanup logic is already quite complex and I'm now
thinking it's not really worth it to add handling this particular case. There's
probably more cases and then if
hantangwangd commented on PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#issuecomment-2316825435
Thank you very much for your guidance @rdblue, it makes sense to avoid users
falling into such a state in advance.
--
This is an automated message from the Apache Git Service.
To
amogh-jahagirdar commented on PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#issuecomment-2316763362
@hantangwangd I thought about this a bit more and while I still think
updating the IncrementalFileCleanup to address this particular case is
possible, I think the question more
rdblue commented on PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#issuecomment-2316382335
@amogh-jahagirdar, @hantangwangd, I'm not sure that incremental cleanup is
doing anything wrong here. Incremental cleanup deletes data files when the
snapshot that removed them from the
RussellSpitzer commented on PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#issuecomment-2314425466
So when I wrote the Spark procedure for this we were already aware that this
code path has a lot of potential issues. We end up basically completely
rewriting the logic of detect
hantangwangd commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1733882501
##
core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java:
##
@@ -327,4 +342,34 @@ private Set findFilesToDelete(
return filesToDelete;
}
+
amogh-jahagirdar commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1733319305
##
core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java:
##
@@ -327,4 +342,34 @@ private Set findFilesToDelete(
return filesToDelete;
amogh-jahagirdar commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1733319305
##
core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java:
##
@@ -327,4 +342,34 @@ private Set findFilesToDelete(
return filesToDelete;
amogh-jahagirdar commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1733319305
##
core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java:
##
@@ -327,4 +342,34 @@ private Set findFilesToDelete(
return filesToDelete;
hantangwangd commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1733280303
##
core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java:
##
@@ -327,4 +342,34 @@ private Set findFilesToDelete(
return filesToDelete;
}
+
hantangwangd commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1733265734
##
core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java:
##
@@ -327,4 +342,34 @@ private Set findFilesToDelete(
return filesToDelete;
}
+
amogh-jahagirdar commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1731784757
##
core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java:
##
@@ -327,4 +342,34 @@ private Set findFilesToDelete(
return filesToDelete;
amogh-jahagirdar commented on code in PR #10983:
URL: https://github.com/apache/iceberg/pull/10983#discussion_r1731788916
##
core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java:
##
@@ -61,17 +63,21 @@ public void cleanFiles(TableMetadata beforeExpiration,
TableMet
19 matches
Mail list logo
