RussellSpitzer commented on PR #15590: URL: https://github.com/apache/iceberg/pull/15590#issuecomment-4185231601
I think this is very interesting, there are a few notes. We should be careful about our target file size of manifests , we are basically ignoring that an potentially just generating a lot of small files. We also previously had a bit of a protection against adding the same file more than once, now that's possible based on the flush threshold so this is a bit different. I think if we fix up the naming, and the flush behavior to be more in line with the manifest target size I think this makes a lot of sense. This code is also going to probably be a big part of the V4 work in the adaptive tree so we should make sure @amogh-jahagirdar takes a look here or we collaborate on this going forwards. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
