vbarua commented on PR #13772: URL: https://github.com/apache/datafusion/pull/13772#issuecomment-2578996616
> With the latest changes, supporting extension tables no longer requires a fork of datafusion, but merely a custom implementation for the new traits. Nice! > However, there's a bit of boilerplate/delegation/copy-paste required for leveraging the default functionality. @ccciudatu do you have an example of any standard and/or built-in DataFusion functionality that can't be serialised to Substrait without this boilerplate? Is it primarily or only UserDefineTypeRels? I'm realising that I don't have a good idea of what this would actually be needed for in standard DataFusion. If there is existing functionality that needs this kind of handling to work, I think it makes sense to include it in the default consumer and producer. Most of my experience with ExtensionTables in Substrait (and hence biases) comes from generating plans outside of DataFusion where we have table-like sources, like embedding a SQL query for another system or fetching data from an internal cache store, which aren't part of standard Substrait (and are also highly-specific to our use case and deployments). For stuff like this we define our own handling because we don't expect DataFusion to be able to handle our custom messages. > a custom TableProvider implementation is hardly aware of the names under which users choose to register the corresponding tables (especially for UDTFs) That's somewhat my argument for not supporting custom stuff generally. The only people who can know what names they used are the ones generating the plans and wiring in the UDTFs in the first place. As long as they have the hooks in place to wire things up how they want, we can service their needs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
