kevinjqliu commented on code in PR #1023:
URL: https://github.com/apache/iceberg-python/pull/1023#discussion_r1709963961
##########
tests/integration/test_writes/test_writes.py:
##########
@@ -1309,3 +1309,27 @@ def
test_table_v1_with_null_nested_namespace(session_catalog: Catalog, arrow_tab
# We expect no error here
session_catalog.drop_table(identifier)
+
+
[email protected]
+def test_overwrite_all_data_with_filter(session_catalog: Catalog) -> None:
+ schema = Schema(
+ NestedField(1, "id", StringType(), required=True),
+ NestedField(2, "name", StringType(), required=False),
+ identifier_field_ids=[1],
+ )
+
+ data = pa.Table.from_pylist(
+ [
+ {"id": "1", "name": "Amsterdam"},
+ {"id": "2", "name": "San Francisco"},
+ {"id": "3", "name": "Drachten"},
+ ],
+ schema=schema.as_arrow(),
+ )
+
+ identifier = "default.test_overwrite_all_data_with_filter"
+ tbl = _create_table(session_catalog, identifier, data=[data],
schema=schema)
+ tbl.overwrite(data, In("id", ["1", "2", "3"]))
+
+ assert len(tbl.scan().to_arrow()) == 3
Review Comment:
ah, I see. The change is to make `delete` a no-op.
Sequence of operation
* pass in `overwrite_filter which matches the entire table
* in `delete`, the `overwrite_filter` is inversed, `preserve_row_filter`
* use `preserve_row_filter` on data files.
* if the result is empty, then we don't include this data file in deletion
Previously, we end up trying to write an empty data file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
