JingsongLi opened a new pull request, #1079: URL: https://github.com/apache/paimon-rust/pull/1079
## Summary Align Native batch incremental changelog planning with the Java implementation and preserve row kinds across the Python row/Arrow write bridge. - Pack DELTA and CHANGELOG manifest lists through the same batch planner. Changelog files from the entire snapshot range are grouped before split packing, worker sharding and LIMIT, instead of planning each snapshot separately. - Support existing bucket/file sharding for combined Changelog and Auto scans. This removes a real PyPaimon Native-plan fallback for a capability already supported by its Python scanner. - Preserve physical INSERT, UPDATE_BEFORE, UPDATE_AFTER and DELETE events, including overlapping key versions, compact-produced changelogs, end-snapshot metadata and empty `(start, end]` boundaries. Keep the per-snapshot incremental API's behavior. - Normalize kindless PK batches to explicit INSERT before buffering. Mixing Arrow batches and row batches with explicit kinds must neither discard DELETEs nor panic during concatenation. - Accept explicit row kinds in cross-partition writes, using the existing migration-delete routing. Treat nullable kind slots as INSERT consistently with ordinary KV writes. No new public API, storage layout, changelog production mode or compaction feature is introduced. The Python bindings remain a thin wrapper over core. ## Java semantics Compared against current Paimon master: - `IncrementalDeltaStartingScanner`: select `(start, end]`, skip OVERWRITE changelogs, include existing COMPACT changelogs, then call `splitForBatch` across the complete range. - `SnapshotReaderImpl`: bucket shards for merged PK files; Java filename-hash shards when files are independently raw-convertible. - `RowKindGenerator` / `TableWriteImpl`: configured `rowkind.field` takes precedence; otherwise preserve the input row kind. - `GlobalIndexAssigner`: preserve input events and migration DELETEs. The related PyPaimon contribution adds end-to-end coverage and repairs Python writer behavior exposed by those tests. Native CI continues to build paimon-rust from main. ## Verification - `cargo test --locked -p paimon --lib table::table_write::tests::`: **71 passed**. - Incremental batch/changelog/timestamp, stream scan, audit log and incremental diff integration tests: **92 passed**. - Python binding read/write tests: **168 passed**. - PyPaimon Native tests with the locally rebuilt extension: **2537 passed, 13 skipped**. - `cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings`: passed. - `cargo +1.98.0 fmt --all -- --check` and `git diff --check`: passed. Independent review caught and verified repairs for the empty-boundary regression, mixed-schema row-kind loss/panic, and nullable cross-partition row-kind routing. New regressions cover these scenarios. No NaN or signed-zero tests were added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
