BsoBird commented on issue #10427:
URL: https://github.com/apache/iceberg/issues/10427#issuecomment-2146405147
> By some means, I assumed that you were hinting at an external service like
a `{REST,Hive,Glue}` catalog. Because the FileIO does not allow list operations
because that doesn't scale for object stores.
The current use of hadoopcatalog/fileSystemCatalog assumes that the file
system provides atomic operations. Object stores do not have this feature.
As long as the file system meets this feature, we can find the latest
metadata and parse out the latest version through the list operation. For
object storage systems, we should avoid fs.rename. We should use fs.write to
commit the metadata directly. This requires a new set of logic. But that's a
different topic.
After the previous PRs are merged, I'd be happy to submit a PR related to
fileSystemCatalog that supports object storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
