stantheman0128 opened a new pull request, #1671: URL: https://github.com/apache/datafusion-python/pull/1671
# Which issue does this PR close? Part of #1577. This PR covers 3 of the 6 items in that umbrella issue (see "Not included" below for why the rest are left out). # Rationale for this change The FFI-pipeline typing audit referenced in #1577 found several places where DataFusion's FFI types are already imported on the Rust side, but the Python surface has no typed `Protocol` describing the expected PyCapsule dunder method. Callers are left to read the Rust source or guess the signature. # What changes are included in this PR? Adds three `Protocol` classes, following the existing `TableProviderExportable` / `PhysicalOptimizerRuleExportable` pattern (name, docstring, and single dunder method), and points the corresponding parameter/return type hints at them: - `datafusion.user_defined.TableFunctionExportable`: describes the `__datafusion_table_function__(self, session)` PyCapsule method already duck-typed via `hasattr` in `TableFunction.__init__` / `TableFunction.udtf` (those `hasattr` checks are the actual runtime dispatch and are left untouched). The `func` parameter on `TableFunction.__init__` is now typed as `Callable[..., Any] | TableFunctionExportable`, and a new `udtf` overload documents the FFI-capsule call shape. - `datafusion.context.ExtensionOptionsExportable`: describes `__datafusion_extension_options__(self)`, required by `SessionConfig.with_extension` (see `crates/core/src/context.rs`, `SessionConfig::with_extension`). `with_extension`'s `extension` parameter is now typed against it. - `datafusion.context.TaskContextProviderExportable`: describes `__datafusion_task_context_provider__(self)`. `SessionContext` already exposes one of these in `context.py`, and this protocol documents the shape for other extensions that want to decode or produce one, matching the `task_context_from_pycapsule` helper already in `crates/util/src/lib.rs`. # Not included from #1577's item list The remaining 3 items are execution-engine-level questions rather than typing changes, so I split each into its own issue for design discussion: - **`FFI_TableProvider`, `TableProvider(ABC)` (item 2):** wiring a plain-Python subclass through to the Rust `TableProvider` trait needs a new `RustWrappedPyTableProvider`. Design discussion: #1668 - **`FFI_TableProviderFactory`, `from_pycapsule` helper (item 5):** the call site this item points at passes an argument to the dunder method, which the existing `from_pycapsule!` / `try_from_pycapsule!` macros don't support (`call0()` only). Discussion: #1669 - **`WindowUDF`, `ABC` equivalent to `Accumulator` (item 6):** `WindowEvaluator` already provides this capability; it's just not `ABCMeta`-shaped because the required `evaluate*` method depends on a flag matrix. Confirming whether the literal ABC form is still wanted: #1670 # Are there any user-facing changes? No runtime behavior changes. This only adds type hints (new `Protocol` classes) and updates existing parameter/return annotations to reference them. Validation run locally: - `uvx [email protected] check python/datafusion/user_defined.py python/datafusion/context.py` - passed - `uvx [email protected] format --check python/datafusion/user_defined.py python/datafusion/context.py` - passed - `git diff --check` - passed - `python -m py_compile` on both files - passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
