NoahKusaba commented on PR #2217: URL: https://github.com/apache/datafusion-ballista/pull/2217#issuecomment-5645898069
> hey @NoahKusaba whats the status of this PR? who do we wait for ? Basically all iceberg-datafusion PR's seem to be blocked over on iceberg-rust until the migration to the new repo is finally done (vote is pending until next week). After that I need to get all of my changes merged in from https://github.com/apache/iceberg-rust/pull/2613, which I'll split up into 3-4 PR's. There is one hitch, which is that iceberg-datafusion maintains two table providers: StaticTableProviders for reads and IcebergTableProvider for everything else. Currently when we do a try_decode_table_provider for the LogicalCodec in the scheduler, datafusion-proto encodes DML write targets as synthetic TableScan nodes, so we don't have a way to distinguish between reads and writes and being able to build the appropriate TableProvider accordingly. mbutrovich suggested making a PR to datafusion to address distinguishing reads from writes in the codec. I put together a prototype that "works", but given that the iceberg-datafusion's situation was in flux I tabled working on it until I could get the bulk of the other iceberg-datafusion PR's in. I'm also not completely satisfied with the solution I came up with either, and it needs more work. You can see the bulk of that discussion over at : https://github.com/apache/iceberg-rust/pull/2862 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
