alexanderbianchi commented on PR #3207: URL: https://github.com/apache/iceberg-rust/pull/3207#issuecomment-5667455682
@Smith-Cruise I'd be interested in your thoughts on the catalog provider + table provider. In https://github.com/apache/iceberg-rust/pull/3000 I realized that there might be some restructuring to do there, which I'll open an issue for once it's moved to `apache/datafusion-iceberg`. Main things that jump out to me - we should probably move to `AsyncCatalogProvider` for a production-ready catalog provider. The current one that caches tables on startup can be kept for OOTB local usecases - We should have a way for table providers to be instantiated with the snapshot they are targeting and not have to call `load_table` in the `scan() - The planning for the ArrowReader should happen in the `Scan()` right now it happens in the data source Datadog also maintains our own CatalogProvider, so hearing your reasons for maintaining your own could help us come up with something prod worthy in the OSS. Ideally with the right extension points for different usecases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
