suxiaogang223 opened a new pull request, #67677:
URL: https://github.com/apache/doris/pull/67677
### What problem does this PR solve?
Add an opt-in native Paimon write path using the official paimon-cpp 0.3.0
SDK, while preserving JNI for unsupported write capabilities. This branch
targets `branch-4.1`.
The native backend is limited to SDK construction and resource injection,
Arrow conversion, commit-message serialization, and SDK shutdown/resource
release. Capability selection stays in FE; storage access uses Doris filesystem
implementations.
### Changes
- Build and package paimon-cpp in an isolated thirdparty prefix. Link
Parquet/Avro registration libraries explicitly without importing the SDK's
private Arrow C++ targets. Validate allocator/runtime dependencies during
thirdparty packaging.
- Add a minimal native Thrift descriptor. Keep EXPLAIN selection reasons in
FE and reuse the existing column-name order.
- Inject a query-attributed SDK memory pool and Doris filesystem adapter.
Retain Arrow conversion resources for their actual buffer lifetime, including
cross-thread release.
- Track task-owned output files until local commit-message handoff. Attempt
cleanup on failure, preserve the original task error, and continue backend
shutdown if abort fails. No separate native writer phase machine.
- Keep final snapshot commit in Java FE through the existing DPCM
commit-message path. Pin SDK creation to the FE-selected schema ID with a small
integration patch.
- Remove the unused Rust FFI backend implementation without reusing its
protocol ID.
- Simplify JNI local lifecycle state and use existing JNI local-reference
wrappers for schema and commit payloads.
### Scope and behavior
- Disabled by default: requires `WITH_PAIMON_CPP=ON` and session variable
`enable_paimon_cpp_writer=true`.
- Initial native scope: unpartitioned append tables, no primary key,
`bucket=-1`, explicit `write-only=true`, Parquet data files, Avro manifests,
and supported primitive columns in table order.
- Storage: local/shared POSIX paths and object stores resolved by Doris
storage providers to the native S3 filesystem. This is not a claim that every
cloud provider has been end-to-end validated.
- Unsupported capabilities select JNI before execution. An already-started
native write is not replayed through JNI after a failure.
- The SDK pool limit is not a cap on all process memory. Cleanup is best
effort; this PR does not claim recovery from arbitrary process OOM or
indefinitely blocked IO. Existing JNI global close-failure policy is unchanged.
### Validation
Completed on the latest local changes:
- `git diff --check` and targeted C++ formatting checks.
- Shell syntax checks and `bash thirdparty/test/paimon-cpp-runtime-test.sh`.
- Configure-only CMake checks using fixture artifacts: Linux/macOS plugin
dependency declarations, native-disabled configuration, and missing-plugin
rejection. No actual native linking was performed by these checks.
- Java/C++ Thrift code generation into a temporary directory with Thrift
0.16.0 to verify the revised interfaces.
Tests added/updated:
- SDK pool allocation failure/accounting and cross-thread release tests.
- Filesystem mapping, output ownership, failed-delete retry and cleanup
tests.
- Writer exception cleanup and commit-message handoff tests.
- FE capability selection and native descriptor Thrift round-trip tests.
- Native plugin registration/serializer-version checks and JNI
repeated-close test.
- Gated MinIO recovery regression covering native commit/readback, prepare
serialization OOM, close-error reporting, message-retention OOM, subsequent
writes and JNI fallback.
**The final revision has not been compiled; C++/Java unit tests and the
MinIO regression have not been run. Build, real plugin linking and runtime
recovery validation remain outstanding. Earlier Fedora validation predates the
latest refactoring and is not presented as validation of this revision.**
### Release note
Add a disabled-by-default native C++ Paimon append writer with FE-side JNI
fallback for unsupported capabilities.
### Check List (For Author)
- Test coverage included (execution status is listed above):
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes. Native writes are explicitly opt-in; JNI remains the default.
- Documentation:
- No documentation files are included in this code PR. User-facing
enablement documentation remains future work before promoting the experimental
path.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases and execution results
- [ ] Confirm documentation requirements
- [ ] Confirm target branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
