plusplusjiajia opened a new pull request, #1103: URL: https://github.com/apache/paimon-rust/pull/1103
### Purpose Linked issue: N/A. Python `SQLContext.sql()` and `sql_with_options()` collect all result batches before returning. Add `sql_stream()` and `sql_stream_with_options()` so callers can consume PyArrow batches incrementally and stop reading early. ### Brief change log - Return the existing `RecordBatchReader` from DataFusion's execution stream, preserving SQL error conversion and PyArrow string compatibility. - Reuse the reader's cancellation support and release its stream on EOF or a read error. - Add regression coverage for multiple batches, Paimon reads, empty results, errors after a successful batch, early close, and `SQLOptions` restrictions. ### Tests - Python: 205 tests passed across `test_sql_stream.py`, `test_datafusion.py`, and `test_read.py`, using CPython 3.10 and locally generated filesystem Paimon fixtures. - `cargo test --locked -p pypaimon_rust --lib`: 92 tests passed, including pending-read cancellation and stream cleanup, using an embedded CPython 3.11 interpreter. - `cargo clippy --locked -p pypaimon_rust --all-targets --no-deps -- -D warnings`: passed. - `cargo fmt --all -- --check` and `git diff --check`: passed. ### API and Format Adds `SQLContext.sql_stream(sql)` and `SQLContext.sql_stream_with_options(sql, options)`, both returning `RecordBatchReader`. Existing readers shared by `TableRead.read_arrow()` and shard updates now release their stream on EOF or an underlying read error. The failing read raises the error; subsequent reads return `None`. Callers must create a new reader to retry. The binding avoids collecting the entire output, while operators such as sorting and aggregation may still buffer data. No storage format changes. ### Documentation Adds Python type stubs and a README example using `contextlib.closing` to release the reader when iteration stops early. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
