suxiaogang223 commented on PR #69886:
URL: https://github.com/apache/doris/pull/69886#issuecomment-6974000665

   # Local self-review after rebase
   
   PR: https://github.com/apache/doris/pull/69886
   
   Head: `7ad7d7890cea93e3aff26a36afea70aac1842aa7`
   Base: `098d0bb62bbc82d9bd999d391efe1c20df89930b`
   
   FE selects Rust/JNI before input, BE writes and prepares files, the report 
acknowledgement transfers ownership, and FE publishes one Paimon snapshot. 
Rebase preserves this invocation chain. The previous FE UT merge conflict is 
resolved by advancing the combined connector surface to API 35.
   
   ## Part 1.3 checkpoints
   
   | Checkpoint | Conclusion |
   | --- | --- |
   | 1. Goal and proof | Rebase the Rust append writer onto master and rerun 
CI. The SPI version test and both surface tests verify the resolved collision; 
Paimon behavior tests exercise writer selection, descriptors and v14 decoding. |
   | 2. Scope and reuse | Range-diff shows only the SPI POM/version assertion 
changes in the first replayed commit; the other eight commits replay unchanged. 
Both upstream path-partition and feature write-diagnostic methods are retained. 
No thirdparty diff remains. |
   | 3. Concurrency and locks | Each pipeline local state owns a writer. The 
admission mutex protects the active count; SDK calls, workload refresh and file 
deletion run outside it. Arrow release on Rust threads restores the query 
context. FE transaction state remains synchronized. |
   | 4. Lifecycle and static initialization | Native stop precedes 
Arrow-pool/budget release. Prepared-file cleanup retains messages, builder and 
tracker until report resolution. Failure exits release admission/credits; 
metrics initialize lazily. These paths are unchanged by rebase. |
   | 5. Dynamic configuration | Both BE limits remain mutable. New writers read 
current values; active writer budgets stay fixed. Online adjustment and retry 
are covered by existing BE tests. |
   | 6. Compatibility | SPI major 35 separates the combined surface from 
upstream 34. Missing descriptors/JNI methods fail safely. Functional 
mixed-version Paimon reads/writes are an explicitly accepted limitation, 
including with Rust writes disabled; no compatibility layer is added. |
   | 7. Thrift/Protobuf IDs | Read all five touched definitions on HEAD, 
master, 4.2, 4.1, 4.0 and 3.1. Common field IDs, names, types and optionality 
agree; all added fields are optional; descriptor ID 7 stays reserved. Upstream 
BLOB fields 21-23 are retained and scan field 43 is free. No protobuf change. |
   | 8. Parallel paths | Rust reads and Rust/JNI writes retain the shared 
pinned descriptor. REST-vended credentials select JNI before input consumption 
and reconstruct refreshable RESTTokenFileIO. Upstream scan API additions are 
preserved. |
   | 9. Conditions and errors | SDK failures become non-OK Status and never 
trigger JNI fallback after input. Wide-row checks use the admitted budget and 
oversized batches advance by one row. No new condition or error-suppression 
path is introduced by rebase. |
   | 10. Test coverage | Existing tests cover null/scalar conversion, wide 
rows, cancellation, admission/credit release, prepare/open failure, report 
ownership, commit framing and REST refresh. Updated SQL cases await CI; no real 
STS expiry or process-crash test is claimed. |
   | 11. Test results | On the rebased head, 1,300 FE/Java tests completed with 
zero failures/errors and one skipped, including all 165 SPI tests. Both 
recorded baselines passed. Prior 59 ASAN BE tests are older evidence for 
unchanged writer sources; current BE and SQL execution awaits CI. Generated SQL 
outputs are unchanged. |
   | 12. Observability | Writer selection, batches/rows, Arrow/native/fixed 
credit and failure/abort timings remain in profiles. Active/credit/rejection 
metrics and bounded FE transaction labels remain intact. No per-row INFO 
logging is added. |
   | 13. Persistence and failover | FE remains the snapshot committer through 
the existing transaction framework; no new EditLog format. 
COMMITTING/OUTCOME_UNKNOWN preserve files. Durable recovery of an in-flight 
external transaction across FE death remains a framework limitation. |
   | 14. Atomicity and crash cleanup | All BE messages feed one FE snapshot 
commit. ACK, definite rejection and ambiguous reports have distinct ownership. 
Abort cleans only unpublished files; process death or deletion failure can 
leave orphans for Paimon cleanup. |
   | 15. FE/BE propagation | Sink/scan descriptors and commit version still 
match BE consumers; JNI uses standard Thrift and includes fe-thrift in its 
isolated plugin. SPI forwarding and the combined baseline agree with API 35. |
   | 16. Performance and memory | 8 MiB/65,536-row batch targets and bounded 
commit exports remain. Arrow is Doris-tracked; native/fixed/metadata credits 
are estimates, not complete RSS enforcement. Spill, revocation and queued 
admission are outside the accepted scope; concurrency admission can fail a 
statement and limits are adjustable online. |
   | 17. Other issues | Reviewed the upstream delta for overlap: new scan SPI, 
BLOB wire definitions, read-only Arrow optimizations and Fluss runtime cache 
initialization. No new conflict with this feature was found. The earlier 
statistics/test_replace_table Jindo missing-class failure also occurs on 
unrelated PRs and remains outside this rebase. |
   
   ## Verification (local)
   
   - VERIFIED: `run-fe-ut.sh --run` selecting Paimon*Test and all 37 SPI 
classes, on 7ad7d7890cea93e3aff26a36afea70aac1842aa7, Maven cache disabled: 
BUILD SUCCESS; 1,300 tests, zero failures/errors, one skipped. All 165 SPI 
tests passed. Log: `output/paimon-write-050/fe-tests-rebase-20261118.log`.
   - VERIFIED: `FE_MAVEN_THREADS=4 MVN_OPT=-Dmaven.build.cache.enabled=false 
./build.sh --fe --be-java-extensions` with the same Paimon-focused extension 
selection as before: successfully built Doris, including Checkstyle and 
packaging/layout validation. Log: 
`output/paimon-write-050/fe-package-rebase-20261118.log`.
   - VERIFIED: range-diff, exact source comparison for Paimon BE/Java paths, 
both SPI baseline tests, and all required branch protocol comparisons passed. 
User runtime configuration diffs are preserved exactly.
   - NOT_RUN locally: BE compilation/tests and SQL regression on this rebased 
tree; these are assigned to the requested CI rerun. Older BE/SQL evidence 
remains explicitly tied to its original head.
   
   Result: **PASS**, one local self-review round after rebase; no unresolved 
findings. This review is not an independent review or a claim that the pending 
CI has passed.
   
   <!-- doris-repo-review:v1:begin -->
   ```yaml
   schema: doris-repo-review/v1
   status: PASS
   pr: apache/doris#69886
   commit: 7ad7d7890cea93e3aff26a36afea70aac1842aa7
   base: 098d0bb62bbc82d9bd999d391efe1c20df89930b
   reviewed_at: 2026-11-18T09:06:27+00:00
   reviewer: suxiaogang223
   model: gpt-6.1-sol
   effort: xhigh
   findings: {blocker: 0, major: 0, minor: 0, nit: 0}
   rounds: 1
   converged: true
   ```
   <!-- doris-repo-review:v1:end -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to