adonm opened a new pull request, #3830: URL: https://github.com/apache/iceberg-python/pull/3830
Related: https://github.com/apache/iceberg-python/issues/271 ## What changes? Honors table sort orders for materialized `pyarrow.Table` writes when every sort field uses an identity transform and one consistent null placement: - sorts unpartitioned tables before bin packing - sorts each final Iceberg partition independently before bin packing - carries the table sort-order ID through `WriteTask` - writes the truthful `DataFile.sort_order_id` Unsupported transforms, nested/missing fields, mixed null placement, and streaming `RecordBatchReader` writes preserve current behavior: data is not claimed as sorted and the file sort-order ID remains null. A warning explains why. ## Why? PyIceberg accepts table sort metadata but currently writes unsorted files and hard-codes `sort_order_id=None`. This prevents readers from safely using sort-order-aware pruning and leaves manifest metadata inconsistent with users' write intent. ## Testing - Unit test verifies physical ascending order, nulls-last behavior, and returned sort-order ID - Integration test verifies scan order and manifest `sort_order_id` - Local InMemoryCatalog append/scan/manifest round trip passed - `PYTHONPATH=. uv run pytest tests/io/test_pyarrow.py -k sort_table_for_identity_sort_order -q` — passed - `ruff check` on changed files — passed - `git diff --check` — passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
