timsaucer opened a new issue, #23678: URL: https://github.com/apache/datafusion/issues/23678
### Is your feature request related to a problem or challenge? At a high level we are trying to expand our FFI crate to support both `datafusion-python` and `datafusion-java` with more extension points. The problem we run into when using these other languages with multiple libraries is that we cannot perform any kind of downcasting of a `&dyn Session` into the underlying `&Session`. There is a discussion of why this is the case on the [datafusion-python online documentation](https://datafusion.apache.org/python/contributor-guide/ffi.html). What we want to do is to expose more of the functions currently in `SessionState` to be usable by anyone with access to a `&dyn Session`. One use case: Users wish to be able to access the `CatalogListProvider` so that a table function could create a new view of an existing table. We cannot currently do this with `&Session` because the `catalog_list()` function does not exist on it. Second use case: Users wish to create a query planner to replace the build in query planner. The problem with this is that the method within `QueryPlanner`, `create_physical_plan()` takes a `&SessionState` parameter. For us to be able to expose a `FFI_QueryPlanner` we need this method to take `&dyn Session`, which in turn needs `PhysicalPlanner` to have a similar change. If we want to create method to expose *or set* a query planner on the `Session` we need to resolve the fact that these traits are defined in `datafusion-core` so not accessible to `Session`. ### Describe the solution you'd like I would like to reorganize our trait definitions. Specifically it seems like we need to either hoist these trait definitions into either `datafusion-session` or the `datafusion-expr` crates. Before doing any work, I think we should review the dependency graph and consider if we should have some thin crates that only expose surface API surfaces or not. On the one hand, that could be cleaner than doing something along the lines of adding `QueryPlanner`, `PhysicalPlanner`, `CatalogProviderList`, `TableProvider`, etc into either `datafusion-session` or `datafusion-expr` crates. The down side of adding additional crates is that we're now adding even more crates to our already pretty large dependency graph. I think the first task should be to review and propose a solution for where the trait definitions should sit. ### Describe alternatives you've considered There is another path where we use type erasure and then downcasting so that we don't have dependency graph problems. Personally I find this to be a hacky way to get around the dependency issue. ### Additional context Related issues: - https://github.com/apache/datafusion/issues/23348 - https://github.com/apache/datafusion/issues/15095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
