comphead commented on PR #17242: URL: https://github.com/apache/datafusion/pull/17242#issuecomment-3240276691
> Here's a diagram that captures the redesign: Thanks @friendlymatthew @xudong963 for diagrams thats incredibly helpful for high level review, couple of questions: - FileOpener should be a separate block? For the reading user it is not clear what is the FileOpener(trait, class, API, etc) - Wondering should we have a sourceconfig to take common parts for MemorySourceConfig and FileSourceConfig? like batch size, etc. Currently MemosySourceConfig implements DataSource which is slightly confusing? - Would be beneficial to show what`TableProvider` provides to FileSource, like table folder/paths, etc - Probably need to reorganize recordbatch block, because as per diagram it doesn't participate in the flow, but the recordbatch it is the exact we waiting from the DataSourceExec -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
