suxiaogang223 opened a new pull request, #65381:
URL: https://github.com/apache/doris/pull/65381
## Summary
Implement Paimon write support for Doris via JNI (Java SDK), with an
abstraction layer design that allows future Rust FFI backend to be plugged in
without upper-layer code changes.
### Architecture
```
VPaimonTableWriter (AsyncResultWriter)
└── IPaimonWriteBackend (abstraction)
├── JniPaimonWriteBackend ← production
└── FfiPaimonWriteBackend ← stub for future Rust v2
```
### Key Features
- **Write path**: Doris Block → Arrow IPC → JNI → Paimon BatchTableWrite
- **Commit path**: prepareCommit() → DPCM-framed byte[][] → FE
PaimonTransaction → StreamTableCommit.filterAndCommit()
- **BE-side bucket routing**: Murmur3-32 hash matching Paimon SDK, splits
Block by bucket, per-bucket concurrent writers
- **INSERT OVERWRITE**: Thrift write_mode + static_partition, Java
withOverwrite() support
- **Type support**: all Paimon primitives + DECIMAL + TIMESTAMP + ARRAY +
MAP + STRUCT
### Changes (12 commits, 34 files, ~5000 lines)
| Layer | Description |
|-------|-------------|
| **Documentation** | Architecture design doc v6.0 (P0 100% coverage) |
| **Thrift** | TPaimonTableSink, TPaimonCommitMessage, RPC wire |
| **BE Abstraction** | IPaimonWriteBackend / IPaimonWriter /
IPaimonCommitter |
| **BE JNI** | JniPaimonWriteBackend (Arrow IPC bridge) |
| **BE FFI Stub** | FfiPaimonWriteBackend (reserved for Rust) |
| **BE Writer** | VPaimonTableWriter + bucket routing + Pipeline Operator |
| **BE Bucket** | Paimon-compatible Murmur3-32 hash (paimon_bucket.h/cpp) |
| **BE RPC** | RuntimeState + PipelineFragmentContext wiring |
| **FE Planner** | Nereids Logical/Physical/Unbound + PaimonTableSink |
| **FE Transaction** | PaimonTransaction + PaimonInsertExecutor |
| **Java** | PaimonJniWriter (Arrow IPC → Paimon SDK) |
### Verified
- ✅ FE Thrift compilation
- ✅ Java paimon-scanner module compilation
- ✅ FE checkstyle: 0 violations
- ⬜ BE compilation (pending CI — local environment unavailable)
### Feature Coverage
P0: 100% | P1: ~80% | Append-only, PK fixed bucket, INSERT INTO, INSERT
OVERWRITE, BE bucket routing all covered.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
