timsaucer opened a new pull request, #1679: URL: https://github.com/apache/datafusion-python/pull/1679
# Which issue does this PR close? Part 3 of 3 in the split of #1672 (part 1: FFI query planner support; part 2: composable extension codecs). Stacked on part 2. #1676 tracks extending this API to additional extension points. # Rationale for this change Working on the Ballista integration showed that chaining the low-level `with_*` methods is easy to get wrong: FFI codecs and planners carry a *weak* task-context provider bound to the context they were created against, so components can end up bound to an intermediate context that is later garbage collected. Queries then fail with `TaskContextProvider went out of scope over FFI boundary`, or worse, silently read stale session state. # What changes are included in this PR? - `SessionContext.with_extensions(*extensions)` installs one or more extension bundles atomically. Each bundle implements the new `__datafusion_session_extension__(ctx)` protocol: it receives the destination context, creates fresh components bound to that exact context, and returns them as a `SessionExtensionComponents` (new public dataclass; `SessionExtensionExportable` is the matching typing protocol). - Internally one destination context is created, factories bind against it, all codecs are installed, and the planner is bound against the final codec chains in a single state write through that context's own `state_ref()`. No context is derived after any provider is created, so every weak provider targets the returned context. - Validation happens before any state change: a failing factory or invalid capsule leaves the source context untouched. - If no bundle supplies a planner, an existing FFI planner on the source context is rebound to the new codec chains; at most one bundle may supply a planner. - The ownership contract is documented and tested: the returned context is the strong owner, and a `DataFrame` outliving it fails with a clean out-of-scope error rather than crashing. - `MyPlannerExtension` in the example crate is a complete Rust implementation of the protocol, including extracting the host's task-context provider from the supplied context. Its codecs record the config value they resolve through the weak provider, letting tests prove the provider targets the returned context rather than the source. - `docs/source/contributor-guide/ffi.md` documents `with_extensions` as the preferred API for extension bundles, keeps low-level chaining as advanced usage, and includes a full three-library registration recipe. # Are there any user-facing changes? New public APIs: `SessionContext.with_extensions`, `SessionExtensionComponents`, and the `SessionExtensionExportable` / `__datafusion_session_extension__` protocol. The context-outlives-DataFrame ownership contract is now documented. No breaking changes to existing APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
