NoahKusaba commented on PR #2862: URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5040802525
> what the benefit of this is vs using the `StaticTableProvider`? > > If we decide this is the right path forward, I wonder if we can make it generic such that this works for incremental append scans etc also? Maybe with a `ScanRange` enum or something? Happy to hear thoughts here! My idea is to register tables to datafusion using IcebergTableProvider, so that the same registration also allows for inserts/deletes/merges, etc. Whereas StaticTableProvider is strictly scan only. Also a staticTableProvider can never go back to current_state as it never refreshes, with my PR unpinning ` with_snapshot_id(None) ` will return back to current state. To make Distributed-Datafusion engines work (in our case it's Ballista), a field like Snapshot_id (and in the future PR catalog_type, name, storage_properties) to this IcebergTableProvider, as the parameters need to round-trips through plan serialization to remote executors. My understanding of how ballista works: Client -> Register Table -> LogicalCodec (serialize/de-serialze provider data) -> Scheduler -> PhysicalCodec (serialize/de-serialize) -> Executors -> Do the thing To reiterate IcebergTableProvider will hold the raw recipe (snapshot_id, table_identity, catalogConfig (to be added in a new PR)), to recreate table connections on distributed executors. It makes slightly more sense in the original datafusion PR I was making (which I'm splitting up) https://github.com/apache/iceberg-rust/pull/2727 Happy to replace snapshot_id with a ScanRange enum as well, we can even replace it in a future PR without breaking anything. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
