timsaucer opened a new pull request, #18568: URL: https://github.com/apache/datafusion/pull/18568
## Which issue does this PR close? This is related to #17374 but does not close it. ## Rationale for this change We have a variety of issues with the current FFI approach - We cannot use functions registered with the session context as expressions. This is a major hindrance to some use cases. For example, if I create a table function that takes an expression as input and I want to use a custom (foreign) function that is not part of the default session context, it will fail during expression deserialization via protobuf. - When we have libraries with complex interplay between local and foreign code we end up wrapping local code in FFI layers when it is not necessary. For example, if I have a foreign catalog provider which produces a local schema provider (which is foreign in the context of the catalog provider) then in my local session context that schema provider is wrapped in a FFI layer even though it uses locally accessible code. - We have unnecessarily library bloat because we have `datafusion` core crate as a dependency, which includes creating full default session contexts. ## What changes are included in this PR? Adds a few new FFI structs: - Function registry - Session - Physical expression Adds in a concept of a library marker ID to determine if we are within a local or foreign context. ## Are these changes tested? Existing unit tests pass and have verified no memory leakage. TODO: - [ ] Add unit test coverage for all new structs ## Are there any user-facing changes? This is a breaking change to the FFI struct. The usage of it changes minimally. TODO: - [ ] Update migration guide to show all required changes - [ ] Add documentation page about the FFI approach, specifically addressing the library marker ID -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
