alexanderbianchi commented on PR #55: URL: https://github.com/apache/datafusion-iceberg/pull/55#issuecomment-6332821513
I think you'd be interested in https://github.com/apache/datafusion-iceberg/pull/67, which has some overlap here. the description goes into more detail. If we build off #67 then the work here should be a lot simpler. I gave it a shot myself https://github.com/alexanderbianchi/datafusion-iceberg-upstream/pull/1 here. I'm personally less convinced by a separate data source, but open to discussion. Questions on seperate data sources: - if we do a data source here, should eager file planning also be it's own data source? - If we do that, does that create 3 data sources eager, sorted eager, and lazy? (x2 for the distributed variants if the design is extended to datafusion distributed) - Who decides the data source? the builder? The table provider? A `match` is going to wind up _somewhere_. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
