suxiaogang223 commented on PR #69886:
URL: https://github.com/apache/doris/pull/69886#issuecomment-6980975406

   # Local self-review after rebase
   
   PR: https://github.com/apache/doris/pull/69886
   
   Head: `fde0e2d56f1a0999020a226f3662672bb88db41c`
   Base: `29c1083f65fe92bb5a778a660fde3e8671ef537b`
   
   FE selects Rust/JNI before input, BE writes and prepares files, the report 
acknowledgement transfers ownership, and FE publishes one Paimon snapshot. 
Rebase preserves this invocation chain. All nine feature patches replay 
unchanged; SPI API 35 is retained. The new upstream delta was checked for 
changes to the APIs and headers this feature uses.
   
   ## Part 1.3 checkpoints
   
   | Checkpoint | Conclusion |
   | --- | --- |
   | 1. Goal and proof | Rebase the Rust append writer onto master and rerun 
CI. The SPI version test and both surface tests verify the resolved collision; 
Paimon behavior tests exercise writer selection, descriptors and v14 decoding. |
   | 2. Scope and reuse | All nine entries in range-diff are unchanged. Paimon 
BE/Java code and tests, the SPI and Thrift definitions are byte-identical to 
the previous head. No new feature code, compatibility layer or thirdparty 
change is added. |
   | 3. Concurrency and locks | Each pipeline local state owns a writer. The 
admission mutex protects the active count; SDK calls, workload refresh and file 
deletion run outside it. Arrow release on Rust threads restores the query 
context. FE transaction state remains synchronized. |
   | 4. Lifecycle and static initialization | Native stop precedes 
Arrow-pool/budget release. Prepared-file cleanup retains messages, builder and 
tracker until report resolution. Failure exits release admission/credits; 
metrics initialize lazily. These paths are unchanged by rebase. |
   | 5. Dynamic configuration | Both BE limits remain mutable. New writers read 
current values; active writer budgets stay fixed. Online adjustment and retry 
are covered by existing BE tests. |
   | 6. Compatibility | SPI major 35 separates the combined surface from 
upstream 34. Missing descriptors/JNI methods fail safely. Functional 
mixed-version Paimon reads/writes are an explicitly accepted limitation, 
including with Rust writes disabled; no compatibility layer is added. |
   | 7. Thrift/Protobuf IDs | Read all five touched definitions on HEAD, 
master, 4.2, 4.1, 4.0 and 3.1. Common field IDs, names, types and optionality 
agree; all added fields are optional; descriptor ID 7 stays reserved. Upstream 
BLOB fields 21-23 are retained and scan field 43 is free. No protobuf change. |
   | 8. Parallel paths | Rust reads and Rust/JNI writes retain the shared 
pinned descriptor. REST-vended credentials select JNI before input consumption 
and reconstruct refreshable RESTTokenFileIO. Upstream scan API additions are 
preserved. |
   | 9. Conditions and errors | SDK failures become non-OK Status and never 
trigger JNI fallback after input. Wide-row checks use the admitted budget and 
oversized batches advance by one row. No new condition or error-suppression 
path is introduced by rebase. |
   | 10. Test coverage | Existing tests cover null/scalar conversion, wide 
rows, cancellation, admission/credit release, prepare/open failure, report 
ownership, commit framing and REST refresh. Updated SQL cases await CI; no real 
STS expiry or process-crash test is claimed. |
   | 11. Test results | At prior head 7ad7d7890ce, 1,300 FE/Java tests 
completed with zero failures/errors and one skipped, including 165 SPI tests; 
Java packaging passed. Prior-head CI FE UT, BE UT and Cloud UT passed. No fresh 
build/UT/SQL run is claimed for this rebase. Current static hygiene, range-diff 
and protocol checks passed. |
   | 12. Observability | Writer selection, batches/rows, Arrow/native/fixed 
credit and failure/abort timings remain in profiles. Active/credit/rejection 
metrics and bounded FE transaction labels remain intact. No per-row INFO 
logging is added. |
   | 13. Persistence and failover | FE remains the snapshot committer through 
the existing transaction framework; no new EditLog format. 
COMMITTING/OUTCOME_UNKNOWN preserve files. Durable recovery of an in-flight 
external transaction across FE death remains a framework limitation. |
   | 14. Atomicity and crash cleanup | All BE messages feed one FE snapshot 
commit. ACK, definite rejection and ambiguous reports have distinct ownership. 
Abort cleans only unpublished files; process death or deletion failure can 
leave orphans for Paimon cleanup. |
   | 15. FE/BE propagation | Sink/scan descriptors and commit version still 
match BE consumers; JNI uses standard Thrift and includes fe-thrift in its 
isolated plugin. SPI forwarding and the combined baseline agree with API 35. |
   | 16. Performance and memory | 8 MiB/65,536-row batch targets and bounded 
commit exports remain. Arrow is Doris-tracked; native/fixed/metadata credits 
are estimates, not complete RSS enforcement. Spill, revocation and queued 
admission are outside the accepted scope; concurrency admission can fail a 
statement and limits are adjustable online. |
   | 17. Other issues | Reviewed the five new upstream commits for overlap. The 
object-storage refactor changes RuntimeState error-log storage and ExecEnv 
storage clients, not the report/budget API used here; this feature references 
none of the removed S3 headers. Other upstream changes concern row-id 
compression, aggregate-state coercion, function hashing and reviewer 
assignment. No additional feature defect found. |
   
   ## Verification and remaining CI work
   
   - VERIFIED (static): all nine feature patches are unchanged in range-diff; 
Paimon writer/reader/Java/SPI/tests and Thrift source comparisons are empty 
against 7ad7d7890ce.
   - VERIFIED (static): build-support/check-build-hygiene.sh passed, and all 
five touched Thrift definitions were compared on HEAD, master and branches 
4.2/4.1/4.0/3.1. No field collision/reuse/new required field.
   - VERIFIED: the user runtime configuration changes exactly match the 
pre-rebase stash; no user docs/native files were changed.
   - Prior evidence, not new runs: 1,300 local FE/Java tests (one skipped), 
Java packaging, and CI FE UT/BE UT/Cloud UT passed at 7ad7d7890ce. CI binary 
compilation also succeeded there; build 6627 subsequently failed in the CI 
Jindo Java launch with E2BIG.
   - NOT_RUN locally on the current rebased tree: compilation, UT and SQL 
execution; these are assigned to the requested CI rerun. The CI helper 
correction reported by the author is to be verified by that run.
   
   Result: **PASS**, one local self-review round after this rebase; no 
unresolved findings in the feature diff. This is not an independent review or a 
claim that pending CI has passed.
   
   <!-- doris-repo-review:v1:begin -->
   ```yaml
   schema: doris-repo-review/v1
   status: PASS
   pr: apache/doris#69886
   commit: fde0e2d56f1a0999020a226f3662672bb88db41c
   base: 29c1083f65fe92bb5a778a660fde3e8671ef537b
   reviewed_at: 2026-11-18T14:08:03+00:00
   reviewer: suxiaogang223
   model: gpt-6.1-sol
   effort: xhigh
   findings: {blocker: 0, major: 0, minor: 0, nit: 0}
   rounds: 1
   converged: true
   ```
   <!-- doris-repo-review:v1:end -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to