JingsongLi opened a new pull request, #1075:
URL: https://github.com/apache/paimon-rust/pull/1075
### Purpose
Related to #807. PyPaimon already supports initial full frames,
single-snapshot delta/changelog frames and bucket-filtered streaming reads. Its
Native adapter currently falls back for L0-visible initial frames, OVERWRITE
changelogs and bucket callbacks, and can resolve a newer snapshot than the
streaming loop selected.
Add the Java per-snapshot planning abstraction to Rust core so those
existing Python operations can use Native planning.
### Brief change log
- Add `ScanMode::{All, Delta, Changelog}` and `SnapshotReader`, available
from `Table` and a configured `ReadBuilder`. `with_snapshot` pins one ID;
`read` does not reinterpret batch startup options. Missing explicitly selected
snapshots fail even with an empty selection or zero limit.
- Reuse the existing manifest and split planner. ALL includes L0 with batch
merge and DV semantics; DELTA/CHANGELOG read one physical manifest list,
including an OVERWRITE changelog. Batch incremental range semantics remain
separate.
- Apply bucket callbacks and `only_read_real_buckets` before split packing
and LIMIT. Keep callback errors and Java shard distribution.
- Keep overlapping L0 keys together for initial DV reads and retain key-only
stats pruning so value predicates cannot resurrect older versions. Keep
materialized vector-search planning behavior intact.
- Preserve physical `_VALUE_KIND` during selected MAP/schema pruning,
alongside `_SEQUENCE_NUMBER`, so projected streaming audit reads receive their
stored row kinds.
- Add a thin Python `SnapshotReader` wrapper, exports and `.pyi`
definitions. The core owns snapshot selection, manifest validation, visibility
and splitting.
Java references: `SnapshotReaderImpl`, `FullStartingScanner`,
`DeltaFollowUpScanner`, `ChangelogFollowUpScanner`, and `DataTableStreamScan`'s
postpone/producer selection rule.
### Tests
- `cargo test --locked -p paimon --lib --test snapshot_reader_test --test
incremental_batch_scan_test --test incremental_timestamp_scan_test --test
incremental_diff_extended_test -- --quiet`: 3,804 unit tests passed (6
ignored), 64 integration tests passed.
- `cargo test --locked -p paimon --test rest_catalog_test query_auth --
--quiet`: 23 passed.
- `cargo +1.98.0 clippy --locked --all-targets --workspace --features
fulltext,vortex -- -D warnings`: passed.
- Rustfmt 1.98.0 and `git diff --check`: passed.
- Python bindings: existing `test_read.py` 131 passed; new
`test_snapshot_reader.py` 16 passed.
- Companion PyPaimon branch with the locally built extension and all five
Native switches enabled: 1,463 passed, 7 skipped, 22 subtests passed; the later
failure/retry regression suite also passed.
- Two independent Java/test review rounds found and repaired the DV packing
and postpone visibility boundaries; projected MAP row kinds and streaming retry
progress are covered by behavioral tests.
### API and Format
Adds the per-snapshot core and Python APIs. No storage format changes or new
dependencies. Query-auth reads on this lower-level path fail closed.
### Documentation
Core API documentation and Python stubs describe snapshot pinning, mode
semantics, bucket selection and errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
