tanmayrauth opened a new issue, #914:
URL: https://github.com/apache/iceberg-go/issues/914
### Feature Request / Improvement
Orphan cleanup (`table/orphan_cleanup.go:428,469`) and snapshot expiration
(`table/updates.go:540`) delete files one at a time via `IO.Remove`. For cloud
object stores this is inefficient — S3's DeleteObjects handles 1000 keys per
call, and GCS/Azure have similar batch APIs.
An optional `BulkRemovableIO` interface should be added to `io/io.go` with
a `RemoveAll(paths []string) error` method. When the IO implementation supports
it, orphan cleanup and snapshot expiration would batch deletes instead of
looping one file at a time.
Implementations that don't support bulk delete simply don't implement the
interface and the existing behavior is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
