JingsongLi opened a new pull request, #10434: URL: https://github.com/apache/paimon/pull/10434
### Purpose Use Native planning for existing PyPaimon streaming capabilities: initial full reads including un-compacted L0, pinned delta/changelog frames (including OVERWRITE changelogs), and bucket-filtered consumers. **Depends on https://github.com/apache/paimon-rust/pull/1075.** This PR stays draft until that core API reaches Rust main, which the Native CI job builds. ### Changes - Route streaming frames through Rust core `SnapshotReader` with an explicit snapshot ID and ALL/DELTA/CHANGELOG mode. Retain the complete structured `read_type`, including nested ROW and selected MAP projections. - Share the existing native split/metadata conversion between batch and per-snapshot planning. Keep manifest selection and split construction in Rust. - Pass bucket callbacks into Native planning and preserve original callback errors. No callback failure is retried by switching to Python planning. - Follow Java `DataTableStreamScan`: hide postpone pending buckets when a changelog producer is configured; keep them visible without one. Apply this rule to Python initial and follow-up fallback paths too. - Apply the bucket filter to Python initial plans, so the same configured consumer reads the same buckets in either backend. - Construct a frame successfully before advancing stream/consumer progress. A failed callback must leave the selected snapshot available for retry; consumer persistence still waits until the yielded frame has been processed. - Count actual Native streaming plans in CI, alongside batch plans. ### Tests After rebasing onto current master, ran 19 relevant suites with all five Native test switches enabled and a local extension built from the companion Rust PR: - **1,467 passed, 7 skipped, 22 subtests passed.** - Native plans: 1,378; reads: 1,748; writes: 1,509; commits: 57. - Native updates: row ID 90, grouped 53, predicate 65, upsert 35, incremental 13, merge 46. - New live tests guard selected-snapshot stability when a later commit exists, multiple overlapping L0/DV files and unsafe value pruning, physical OVERWRITE changelogs, every bucket selection, empty plans, original callback exceptions, projection/predicate dependencies, nested ROW/MAP outputs, zero-column row counts, postpone producer semantics, and actual async stream retry/consumer progress. - Native cases block Python planning and reading fallback to prove the core path was exercised. - CI-configured Flake8, Python 3.6 AST syntax checks and `git diff --check`: passed. Java behavior was checked against `SnapshotReaderImpl`, `FullStartingScanner`, `DeltaFollowUpScanner`, `ChangelogFollowUpScanner`, and `DataTableStreamScan`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
