suxiaogang223 opened a new pull request, #69886:
URL: https://github.com/apache/doris/pull/69886
### What problem does this PR solve?
Problem Summary:
Add an opt-in Rust writer for Paimon INSERT on branch-4.2. Upgrade the
released paimon-rust dependency to v0.4.0, keeping the existing Rust 1.94.0
build requirement unchanged.
- Select Rust only for main-branch, unpartitioned append-only tables with
bucket=-1, Parquet, and supported scalar columns. Unsupported tables retain the
existing JNI writer before any input is consumed.
- Send a lightweight, pinned table descriptor to BE. Rust prepares files and
serializes Java-compatible v11 commit messages; the existing Java FE
transaction path remains responsible for snapshot commit.
- Admit writers using a fixed estimated query memory credit, a BE-wide
concurrency cap, and bounded rows/input batches. Await native shutdown before
releasing Arrow buffers, credit, and admission slots. Abort cleans up partial
and prepared files.
- Maintain a thirdparty patch for the required C interfaces, Java protocol
compatibility, synchronous file rolling, and flushing retained raw input before
exhausting its budget.
The memory credit is an estimate, not a complete Rust allocator or RSS
limit. This MVP does not add spill, memory revocation, live budget resizing, or
interruption of a native call. S3 support requires explicit static credentials
and a region.
Network access: when enabled, BE can issue Rust/OpenDAL requests to the S3
endpoint configured in the existing Paimon catalog. This requires the existing
catalog configuration privileges to choose the endpoint and table INSERT
privileges to initiate a write; this PR adds no artifact download or dynamic
loading path.
### Release note
Add experimental Paimon Rust append writes behind
`enable_paimon_rust_writer` (default `false`). Supported tables can use a
native writer while continuing to commit snapshots through the existing Java FE
path. Add mutable BE settings `paimon_rust_writer_memory_limit_bytes` (64 MiB)
and `paimon_rust_writer_max_concurrency` (8), applied when admitting new
writers.
### Check List (For Author)
- Test
- [x] Regression test: `test_paimon_rust_write_append`, expected output
generated by the runner and verified in a normal rerun. Covers INSERT SELECT,
null/decimal/date values, input and budget failures, and partitioned-table JNI
fallback. Doris and Spark both read 20,003 rows with matching aggregates.
- [x] Unit Test: 10 new BE tests under ASAN, followed by five shuffled
iterations (50/50 passed); 23 focused FE tests; 3 focused Rust
protocol/retained-input tests.
- [x] Manual test: patched SDK build, full ASAN BE build, FE build
including Checkstyle; two Rust writers -> Java v11 decode -> snapshot commit ->
read six rows on local storage and MinIO; prepared/partial-file abort checks.
- [ ] No need to test or manual test.
Formatting: clang-format 16 preset scripts passed. Clang-tidy reported no
new diagnostics in modified files; an existing unmatched NOLINTEND in
`be/src/core/types.h:578` prevents a clean full result. BE tests used the
standard `run-be-ut.sh` entry point with temporary local test-source selection;
the selection change is not part of this PR.
- Behavior changed:
- [ ] No.
- [x] Yes. Opt-in writes can use Rust for the supported append-only
subset. The default remains JNI.
- Does this need documentation?
- [ ] No.
- [x] Yes. User documentation for the experimental switch and supported
subset is pending; keep this PR as a draft.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
