timsaucer opened a new pull request, #1672: URL: https://github.com/apache/datafusion-python/pull/1672
# Which issue does this PR close? Related to #1612. This PR does not close it, but provides the FFI query planner and codec plumbing that a `datafusion-distributed` integration can build on. # Rationale for this change Extension libraries (for example distributed execution engines) need to supply their own `QueryPlanner` to a `SessionContext` without compiling against the `datafusion-python` crate. This PR exposes the query planner over the FFI boundary, following the same PyCapsule pattern used for table providers and catalogs. Supporting a foreign planner also surfaced a codec problem: a query can involve three independent native libraries (datafusion-python, a provider library, and a planner library), and each library needs its extension codecs active on the session at the same time. Previously, installing a logical or physical extension codec replaced the prior codec, so the second library's install silently discarded the first — plans then failed later with a confusing decode error. Codecs now compose. # What changes are included in this PR? **FFI query planner** - `SessionContext.with_query_planner(planner)` installs a planner exported via a `__datafusion_query_planner__` PyCapsule, preserving existing session state and codec settings. - `SessionContext.__datafusion_query_planner__()` exports the current planner so another planner can wrap it as an explicit fallback (a session holds exactly one planner; layering is explicit delegation). - A `RuntimeAwareQueryPlanner` adapter binds foreign planners to the Tokio runtime owned by datafusion-python. - New example crate `datafusion-ffi-query-planner-example` demonstrating a real three-library plan exchange (host, provider library, planner library as separate cdylibs), including session config transfer via `SessionConfig.with_extension`. **Composable extension codecs** - `with_logical_extension_codec` / `with_physical_extension_codec` now prepend to a codec chain instead of replacing the prior codec. The most recently installed codec is consulted first, falling through codec by codec to DataFusion's default codec. A codec signals "not mine" by returning an error. - Encoding runs each codec against a scratch buffer so failed attempts leave no partial bytes, and treats Ok-with-no-bytes (encode by name) as "no opinion" so later codecs still get a chance. - When every codec in the chain fails, the errors are aggregated so the owning codec's diagnostic is not masked by the default codec's generic error. - Fixed a latent bug where installing a codec silently reset `python_udf_inlining` back to enabled. **Documentation and tests** - `docs/source/contributor-guide/ffi.md` gains sections on composable codecs (family-prefix discipline, registration order between libraries no longer matters) and planner layering (install all codecs before exporting or chaining planners, since a planner capsule captures the codecs at export time), including a full three-library registration recipe. - Rust unit tests for chain dispatch semantics; Python integration tests for codec composition on both layers plus an end-to-end three-library test with composed codecs under a foreign planner. # Are there any user-facing changes? Yes: - New APIs: `SessionContext.with_query_planner` and `SessionContext.__datafusion_query_planner__`. - Behavior change: `with_logical_extension_codec` / `with_physical_extension_codec` now compose with previously installed codecs instead of replacing them. Sessions that install a single codec are unaffected. - These methods no longer reset the `with_python_udf_inlining` setting. - New example package `datafusion-ffi-query-planner-example` in the examples folder (not shipped in the wheel). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
