gabotechs opened a new pull request, #26555: URL: https://github.com/apache/datafusion/pull/26555
## Which issue does this PR close? Follow-up to #24631 and #24628. No separate issue. ## Rationale for this change Custom `DataSource` implementations can encode themselves through `try_to_proto`, but still need a `PhysicalExtensionCodec` to decode. Applications combining sources from multiple libraries therefore have to compose their codecs. Extend the session-scoped decoder registry to sources so each source can own its decoder and register it by name, following the extension execution-plan and expression APIs. ## What changes are included in this PR? - Add `ExtensionDataSourceFromProto`, `register_data_source`, and `extension_data_source_node`. Constructors receive a typed `PhysicalExtensionDataSourceNode` and return `Arc<dyn DataSource>`; DataFusion supplies the `DataSourceExec` wrapper. - Add a source-specific wire variant and registry dispatch. Source names are independent of plan and expression names. - Preserve the existing lookup policy: registered decoders take precedence, their errors are final, and unnamed or unregistered nodes fall back to the codec. Existing codec-encoded sources keep their wire format. ## What is the testing strategy for this PR? Six registry unit tests and twelve integration tests cover registration, execution after decoding, reading-session state, codec fallback and precedence, malformed payloads, error-kind preservation, shared names across extension kinds, and binary/JSON round trips. Passed locally: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - Full `datafusion-datasource` and `datafusion-proto` test suites with source proto support and JSON enabled: 539 unit/integration tests and 14 doctests passed. - Protobuf regeneration and formatting of the added upgrade-guide section. The optional `dev/rust_lint.sh` runner could not start because the system Python lacks PyYAML. The full workspace test suite was not run. ## Are there any user-facing changes? New source-registry APIs behind the existing `proto` feature. Built-in source serialization is unchanged. The new `PhysicalPlanType::ExtensionDataSource` variant requires updates to exhaustive enum matches. Readers must be upgraded and source decoders registered before writers emit the new variant. Documented in the 56.0.0 upgrade guide. AI-assisted implementation with Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
