suxiaogang223 opened a new pull request, #68825:
URL: https://github.com/apache/doris/pull/68825
### What problem does this PR solve?
Issue Number: None
Related PR: #66227
Problem Summary:
Forward-port the Paimon Rust reader from branch-4.1 to master. The reader
consumes FE-planned Paimon splits in BE through the Arrow C Data Interface and
is guarded by the `enable_paimon_rust_reader` session variable, which remains
disabled by default.
The forward-port is adapted to the master connector framework instead of
copying the branch-4.1 FE scan path. FE selects the Rust reader through the
Paimon connector scan provider and checks BE capability advertised by
heartbeat. Unsupported table layouts, scan modes, credentials, schemas,
projections, predicates, or runtime filters conservatively fall back to the
existing JNI reader.
Master-specific compatibility fixes include:
- preserve null-aware predicate and runtime-filter semantics;
- reject unsupported credential combinations without changing effective
catalog credentials in tests;
- follow master branch and snapshot resolution behavior;
- carry name-based nested projection state even when a connector has no
stable nested field IDs, preventing projected Arrow structs from being decoded
as complete Doris structs;
- explicitly enable TIMESTAMP_LTZ mapping in the time-zone regression
fixture.
### Release note
Add the experimental `enable_paimon_rust_reader` session variable. When
enabled and supported by the selected BE and table scan, Doris can read Paimon
data through the BE-side Rust reader. Unsupported scans automatically use the
existing JNI reader.
### Check List (For Author)
- Test:
- FE build and checkstyle
- FE unit tests: `PaimonRustReaderSelectorTest`,
`PluginDrivenScanNodeProjectionTest`
- Regression tests: `test_paimon_rust_reader_cast_predicates`,
`test_paimon_rust_reader_schema_evolution`,
`test_paimon_rust_reader_compatibility`, `test_paimon_rust_reader_eq_for_null`
- Behavior changed: Yes. The new reader can be enabled explicitly; the
default remains unchanged.
- Does this need documentation: Yes. Session-variable documentation should
be added with the feature rollout.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
