zaynt4606 opened a new pull request, #10698: URL: https://github.com/apache/paimon/pull/10698
### Purpose Fix limit handling and reader cleanup in `RayDatasource`. The task can emit fewer rows than its `BlockMetadata.num_rows` reports: for example, a five-row split with a scan limit of two still advertises five rows. Ray's `per_task_row_limit` is also passed only to `ReadTask`, without constraining the underlying Paimon reader. Finally, the worker does not explicitly close its Arrow batch reader. - Combine the Paimon scan limit and Ray's per-task row limit before constructing the worker reader. - Cap known task row counts at the effective limit, preserving unknown counts for filtered or primary-key reads. - Close batch readers in `finally`, including normal completion, empty reads, failures, and early generator closure. - Add focused regression tests and strengthen the public API test to check exact counts before and after materialization across two partitions, including zero and oversized limits. This change does not alter public API signatures or catalog authentication behavior. ### Tests Tested against community Paimon master with official Ray wheels, Python 3.12, and PyArrow 19.0.1: - Before the fix, the 14 new datasource cases reproduce **11 failures and 3 passes** on Ray 2.58.0. - After the fix, **69 passed** on Ray 2.58.0 across `ray_datasource_test.py`, `ray_integration_test.py`, `ray_sink_test.py`, and `ray_data_test.py`. - **30 passed on each of Ray 2.44.0, 2.48.0, 2.50.1, and 2.53.0**: the datasource regressions, the exact-limit integration test, and `RayDataTest`. - Changed Python files pass repository-configured Flake8; the Python license-header check and `git diff --check` pass. The new regression tests cover limit combinations, task serialization, unknown filtered counts, reader-level limit pushdown, and batch-reader cleanup. The integration tests use a local Ray cluster and filesystem catalog; remote storage and the optional Rust backend were not exercised. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
