steveloughran commented on code in PR #13084:
URL: https://github.com/apache/iceberg/pull/13084#discussion_r2115891476
##########
api/src/main/java/org/apache/iceberg/actions/DeleteOrphanFiles.java:
##########
@@ -142,6 +142,8 @@ default DeleteOrphanFiles equalAuthorities(Map<String,
String> newEqualAuthoriti
interface Result {
/** Returns locations of orphan files. */
Iterable<String> orphanFileLocations();
+
+ long orphanFileLocationsCount();
Review Comment:
comment?
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -233,46 +243,83 @@ private String jobDesc() {
return String.format("Deleting orphan files (%s) from %s",
optionsAsString, table.name());
}
- private void deleteFiles(SupportsBulkOperations io, List<String> paths) {
- try {
- io.deleteFiles(paths);
- LOG.info("Deleted {} files using bulk deletes", paths.size());
- } catch (BulkDeletionFailureException e) {
- int deletedFilesCount = paths.size() - e.numberFailedObjects();
- LOG.warn("Deleted only {} of {} files using bulk deletes",
deletedFilesCount, paths.size());
- }
+ private SerializableConsumer<Iterator<String>>
bulkDeleteFiles(SupportsBulkOperations io) {
+ return stringIterator -> {
+ List<String> paths = Lists.newArrayList(stringIterator);
+ try {
+ io.deleteFiles(paths);
+ } catch (BulkDeletionFailureException e) {
+ int deletedFilesCount = paths.size() - e.numberFailedObjects();
+ LOG.warn("Deleted only {} of {} files using bulk deletes",
deletedFilesCount, paths.size());
Review Comment:
consider retrying on the failures for 500 and 503
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
