JingsongLi opened a new pull request, #1183: URL: https://github.com/apache/paimon-rust/pull/1183
## What changes were proposed in this pull request? Move the Native split execution that currently lives in PyPaimon into Rust Core. `bindings/python` only passes execution options to `TableRead`; it does not introduce a separate Python executor API. - Add `TableRead::with_parallelism` and `with_preserve_split_order`, and keyword arguments on the existing Python `read` / `read_arrow` methods. - Execute contiguous, byte-balanced split groups with one pending read or ready batch per group. Owned Tokio tasks preserve parallel synchronous Parquet decoding, while single-group reads stay inline. Apply a single output row quota and cancel every group on close, error, or quota exhaustion. - Share the Parquet resource budget. Ordered groups disable background row-group prefetch to avoid a later buffered decoder retaining the permit needed by an earlier group. - Preserve eager schema, configuration and complete authorization-grant validation before grouping. - Keep finite reads that resolve BLOB payloads or REST view descriptors sequential, and divide the total BLOB concurrency budget across workers. Append, primary-key and data-evolution readers retain their existing filtering, merge and row-kind semantics. Incremental diff, audit and format-table readers keep their existing execution pipelines. No dependencies are added. ## Why are the changes needed? PyPaimon currently owns a second execution implementation for Native reads, including worker pools, queues, split grouping and global LIMIT handling. Core should own that behavior so the language bindings can remain adapters. ## How was this patch tested? - Executor unit tests for ordered / ready-first output, bounded prefetch, global LIMIT, cancellation, worker error/panic, runtime requirements and simultaneous synchronous decoding on two runtime worker threads. - Existing `table::table_read::tests`, including restricted authorization and eager-validation cases. - `parallel_table_read_test`: Append / primary-key / data-evolution matrix, filters and limits, row kinds, zero-column output, missing later BLOB payloads, and a real multi-row-group Parquet regression with controlled I/O ordering and a shared byte budget. - Rust binding unit tests invoke both Python methods with execution keywords, including empty splits and invalid parallelism. - Rust 1.98.0 formatting and Clippy with `-D warnings` for the affected Core and binding targets. - Added binding integration tests for the same existing APIs. These Python integration cases require CI: the local Native extension was neither rebuilt nor installed. The matching PyPaimon change delegates Native scheduling to this Core implementation; its CI continues to use Rust main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
