gabotechs opened a new pull request, #26807: URL: https://github.com/apache/datafusion/pull/26807
## Which issue does this PR close? Stacked on #26555; merge that PR first. No separate issue. The branch includes the rebased DataSource change. [Sink-only diff](https://github.com/gabotechs/datafusion/compare/774dfd5d62...codex/datasink-proto-registry). The upstream PR targets `main` because GitHub cannot use a fork branch as the base of an upstream PR. ## Rationale for this change Custom `DataSink` implementations can encode themselves through `try_to_proto`, but decoding still requires an application-owned `PhysicalExtensionCodec`. Sinks from independent libraries therefore need codec composition. Give sinks the same session-scoped, per-type decoder registration as sources in #26555. ## What changes are included in this PR? - Add `ExtensionDataSinkFromProto`, `register_data_sink`, `data_sink_names`, and `extension_data_sink_node` alongside the source APIs. - Register `DataSinkExec` as the shared extension-plan decoder. Its typed payload contains the sink name, sink bytes, and optional sort requirements; the existing extension-plan node carries exactly one input plan. DataFusion reconstructs the input, ordering, and execution-plan wrapper around the decoded sink. - Keep sink names separate from source, plan, and expression names. Registered-wrapper failures are final; an unregistered wrapper retains the existing plan-codec fallback. Built-in and existing codec-owned sinks retain their wire formats. No new top-level plan variant is added. ## What is the testing strategy for this PR? Five registration unit tests and twelve integration tests cover receiving-session state and actual writes after a binary round trip, input and ordering preservation, independent sink types, name isolation, invalid payloads/inputs/orderings, codec dispatch boundaries, legacy codecs, and JSON serialization (with the `json` feature). Validation: 570 affected-crate tests and 14 doctests passed. Formatting, workspace Clippy with warnings denied, rustdoc with warnings denied, protobuf regeneration, and documentation formatting completed. The broader `dev/rust_lint.sh` suite could not start because `/usr/bin/python3` lacks PyYAML. ## Are there any user-facing changes? New opt-in sink registration APIs behind the existing `proto` feature, documented in the 56.0.0 upgrade guide. Register sinks on every reader before writers emit the new wrapper payload. Existing built-in sink formats are unchanged. AI-assisted implementation with Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
