rdblue commented on issue #1398: URL: https://github.com/apache/iceberg/issues/1398#issuecomment-682837392
Our S3 file system doesn't do the `getFileStatus` check when overwriting, which is how we avoid this. I think the best way to handle this is to add a certain amount of retries at the `FileIO` layer. We already have retries for `table.refresh()` (in `BaseMetastoreTableOperations`) that take care of most of these issues for the root metadata. The reason why we added the retries there was to avoid adding extra latency when there actually is a problem -- when a data file is missing, for example. But, it looks like it would be better to have retries for at least metadata files when using S3. I think it makes sense to add retries to `HadoopInputFile` and to make the number of retries and total retry timeout configurable either in table properties or Hadoop Configuration. The downside to adding retries there is that we'd have retries for data files as well, but that seems like a reasonable trade-off to me. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
