daviddallakyan2005 opened a new pull request, #3805: URL: https://github.com/apache/iceberg-python/pull/3805
Closes #3270 # Rationale for this change Index equality delete files and apply them on PyArrow scans so CDC and merge-on-read tables can be read. Equality deletes still cannot be written. This ports the DeleteFileIndex work from #3285 onto current main, skips bounds prune when an equality field has been dropped, and left anti joins matching equality delete rows after positional deletes. It supersedes #3285, which is open and conflicting. Credit to Alex Stephen for the index. Please close #3285 if this path is preferred. When a data file has equality deletes, the scan concatenates that file’s record batches, aligns each matching equality-delete table by field id then concats, adds is_null / fill_null (and is_nan for floats) helper columns per key, left anti joins once, then yields the survivors; files with only position deletes still yield each scanner batch. ## Are these changes tested? Yes. `make test` 3962 passed, 3 skipped, 1570 deselected `make lint` All prek hooks Passed `make test PYTEST_ARGS="-v -k equality"` 168 passed, 5367 deselected `uv run python -m pytest tests/table/test_delete_file_index.py tests/io/test_pyarrow.py tests/catalog/test_scan_planning_models.py tests/table/test_validate.py tests/table/test_init.py -m "(unmarked or parametrize) and not integration" -v` 371 passed, 57 deselected `make test-integration PYTEST_ARGS="-v --tb=short"` 1350 passed, 71 skipped, 3 failed, 4111 deselected The three integration failures match main without this diff. `test_read_spark_written_puffin_dv` raises DeprecationWarning for `puffin.to_vector()`. Two DuckDB tests failed to download the iceberg extension. ## Are there any user-facing changes? Yes. Tables that already store equality deletes can be scanned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
