timsaucer opened a new issue, #1707: URL: https://github.com/apache/datafusion-python/issues/1707
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Writing a datafusion-python extension library means getting a handful of non-obvious lifetime and identity rules right, and there is currently no guidance an agent working in a *downstream* repo can find. The rules are spread across `docs/source/contributor-guide/ffi.md`, the docstrings on `SessionContext.with_extensions` / `with_logical_extension_codec`, `.ai/skills/ffi-capsule-protocol/SKILL.md`, and the two example crates. Each of those is written for someone already inside this repository. The people who need it are not. Ballista and datafusion-distributed each build a separate cdylib plus Python glue against this project's FFI protocol, and they hit the same set of traps that #1679 was opened to fix: components bound to an intermediate context that is later collected, `TaskContextProvider went out of scope over FFI boundary`, codec ids that do not survive a process boundary, and `write_parquet` failing across the FFI codec because it has no file-format hooks. Every one of those is knowable in advance and none of them is discoverable from where those authors are standing. Raised by @ntjohnson1 in review of #1679: *"I wonder if it makes sense to have a todo for some datafusion python extension skill/s ... I do suspect it will require some guidance to make sure they are doing it safely."* **Describe the solution you'd like** An agent skill for extension *authors*, plus a contributor-side audit skill that keeps it from rotting. *Location.* A new `skills/datafusion_python_extension/SKILL.md`, sibling to the existing `skills/datafusion_python/SKILL.md`. Not `.ai/skills/` — that tier is for agents working on this repository, and this audience is working on a different one. `.ai/skills/ffi-capsule-protocol/SKILL.md` stays where it is as the review trigger for changes to the capsule getters themselves, and the two cross-reference rather than duplicate. *Open question this issue needs to settle: how does a downstream agent get the file?* `skills/` is not currently included in the wheel, so an agent in a repo that depends on `datafusion` cannot see it. Options, roughly in order of how well they match "keeping it up to date shouldn't be too bad": 1. Ship `skills/` as package data in the wheel. Downstream gets the current version with the dependency they already have, and it tracks the release they are actually building against. 2. Publish to agentskills.io, the standard `CLAUDE.md` already points at, so `find-skills` discovers it. 3. Document that downstream repos should vendor a copy. Cheapest to do, fastest to go stale. *Content.* Drawn from the traps that #1672 and its three split PRs actually surfaced, not invented advice: - The `__datafusion_session_extension__` bundle protocol: create fresh components on every call, never cache bound components, never retain the context that was passed in. - Take the task-context provider off the supplied context. Never construct a `SessionContext` inside an extension library. (Link `ffi-capsule-protocol`; do not restate it.) - Codec identity: the class-derived id, `__datafusion_codec_id__` for pinning one, and `anon:<uuid>` for a bare `PyCapsule`. A plan that has to decode in another process requires a declared id — this is the rule a distributed engine hits first. - Ownership: the context returned from `with_extensions` is the strong owner. A `DataFrame` does not keep it alive. `TaskContextProvider went out of scope over FFI boundary` is the symptom; holding the context is the cure. - Prefer `with_extensions` to hand-chaining `with_logical_extension_codec` / `with_physical_extension_codec` / `set_query_planner`, and say why. - Known gap: `FFI_LogicalExtensionCodec` has no vtable entries for file formats, so `df.write_csv` / `write_parquet` / `write_json` fail across the boundary. Compose the FFI codec with a `DefaultLogicalExtensionCodec` fallback on `try_encode_file_format` / `try_decode_file_format`. *Sequencing.* This should land after the codec-id gap in `with_extensions` is resolved. `SessionExtensionComponents` currently has no way to declare an id for a codec handed over as a bare capsule, and `with_extensions` takes no `codec_id=`, so the only workaround is a Python shim that declares `__datafusion_codec_id__`. Writing the skill now would document that workaround as the intended design and require a rewrite when the API gains a way to express it. *Keeping it current.* An `.ai/skills/audit-extension-skill/` contributor skill following the existing `.ai/skills/audit-skill-md/` pattern: diff the skill's claims against the two example crates and the current public API, and report drift. This is the part worth automating, because the failure mode is the skill silently describing a protocol that has since moved. **Describe alternatives you've considered** *Extend `.ai/skills/ffi-capsule-protocol` instead of adding a skill.* Wrong audience and wrong trigger. That skill fires when someone touches a `__datafusion_*__` getter in this repository. An extension author is not touching those; they are implementing against them, and they need a build-it-from-scratch narrative rather than a review checklist. *Extend `docs/source/contributor-guide/ffi.md` instead.* Already the reference material, and #1679 adds substantially to it. But it is organized around the protocol rather than around the task, and it lives at a URL a downstream agent has no reason to fetch. The skill should point *at* `ffi.md` for depth rather than replace it. *Assert in CI that the skill can regenerate the three-library example.* This was the original suggestion and it is the right instinct — the example is the best available proof the guidance is sufficient — but it does not survive contact with a workflow. Generated output varies between runs, so a byte-comparison fails on noise and a fuzzy comparison asserts nothing. The deterministic half is the audit skill above. The full generate-from-skill-and-run-the-tests exercise is real value in the wrong shape for CI; it belongs on the release checklist as a manual step. **Additional context** Reference implementations the skill should point at, both exercised in CI via `.github/workflows/test.yml`: - `examples/datafusion-ffi-example/` — table providers, UDFs, catalog providers, optimizer rules, extension codecs. - `examples/datafusion-ffi-query-planner-example/` — query planner, and `MyPlannerExtension` in `src/extension.rs` as a complete Rust implementation of `__datafusion_session_extension__`. Follows up on #1679 (part 3 of the #1672 split). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
