karuppayya commented on PR #13084: URL: https://github.com/apache/iceberg/pull/13084#issuecomment-2956602599
Thanks @RussellSpitzer for the feedback! I agree that exposing the DataFrame of orphan files would be useful that would give users the granular control to perform custom logic on those files. I can add a new API for it.(and see if it can be used for `dry-run` instead of a no-op delete). For users who simply want to remove orphan files action /procedure, distributed delete would be a good improvement(especially at scale). It avoids collecting [all file paths on the driver](https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java#L257-L258)(before the deletion), which can become a bottleneck. This change tries to handle the deletion in a [distributed manner](https://github.com/apache/iceberg/pull/13084/files#diff-bd340e00ffa7b90d78d80524d654a754b0d5f7454aa7fd96da43ec19174209a9R284), making the process efficient. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
