RussellSpitzer commented on PR #13084: URL: https://github.com/apache/iceberg/pull/13084#issuecomment-2956675423
I understand the proposed changes but I'm not a big fan of the API and behavior changes required to make it happen within our current framework. It seems like the kind of thing we should do in a Iceberg V2 Library rather than a modification of the existing code. For example the "Cache" and "count' pattern here would be make this much more expensive for smaller remove orphan files counts but this can't be opted out of. I'm not sure why we are hard coding in coalesce 10 , etc... And basically any time we have to do an accumulator in Spark i'm suspicious. I feel like a lot of these changes are just to try to match old behaviors. So the question is are these changes better than having the user do something like orphanFiles().mapPartitions(DeleteOprhanFilesSparkAction.distributedDeleteFunction).collect -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
