gabeiglio commented on issue #3162: URL: https://github.com/apache/iceberg-python/issues/3162#issuecomment-4322408580
Frequently committed to tables can increase in metadata size rapidly due to snapshots (specially with partition summaries on). Iceberg Java has implemented a catalog property called Refs-only loading that allows IRC to only send the table metadata with currently referenced snapshots (main branch current snapshots, and any tags or branches) and only load historical snapshots as needed. [Here](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java#L527C38-L527C42) is a reference implementation from the Java repo. This would be very valuable to have in PyIceberg :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
