yangshangqing95 opened a new issue, #17439:
URL: https://github.com/apache/iceberg/issues/17439
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
The `entries` and `all_entries` metadata tables will return wrong results
when filtering `data_file.content` with negative predicates such as `!=` or
`NOT IN`.
Both metadata tables use `BaseEntriesTable.ManifestContentEvaluator` to
prune manifests before reading their entries.
The evaluator currently maps file-level content values to manifest-level
content values as follows:
```text
FileContent.DATA -> ManifestContent.DATA
FileContent.POSITION_DELETES -> ManifestContent.DELETES
FileContent.EQUALITY_DELETES -> ManifestContent.DELETES
```
This mapping is safe for positive predicates such as `=` and `IN`, where the
evaluator only needs to determine whether a manifest may contain a matching
entry.
However, it is not sufficient for negative predicates.
For example, consider the following predicate:
```sql
data_file.content != 2
```
Here, `2` represents `FileContent.EQUALITY_DELETES`.
When evaluating a delete manifest, the current implementation maps
`FileContent.EQUALITY_DELETES` to `ManifestContent.DELETES`. It then concludes
that the delete manifest cannot contain matching rows and prunes the entire
manifest.
However, the delete manifest may contain position delete files whose content
value is:
```text
1 = FileContent.POSITION_DELETES
```
Those entries satisfy:
```text
1 != 2
```
and must be returned.
This causes false-negative pruning and incomplete query results.
## Affected metadata tables
Both metadata tables are affected because they delegate manifest planning to
`BaseEntriesTable.planFiles()`:
```text
<table>.entries
<table>.all_entries
```
## How to reproduce the issue
Create a format v2 table configured to use merge-on-read deletes:
```sql
CREATE TABLE local.demo.entries_content_bug (
id INT,
data STRING
)
USING iceberg
TBLPROPERTIES (
'format-version' = '2',
'write.delete.mode' = 'merge-on-read',
'write.delete.granularity' = 'file',
'write.distribution-mode' = 'none',
'write.target-file-size-bytes' = '536870912'
);
```
Insert multiple rows into a single or small number of data files:
```sql
INSERT INTO local.demo.entries_content_bug
SELECT /*+ REPARTITION(1) */
CAST(id AS INT),
CONCAT('value-', CAST(id AS STRING))
FROM RANGE(1, 1001);
```
Delete one row:
```sql
DELETE FROM local.demo.entries_content_bug
WHERE id = 500;
```
Now run:
```sql
SELECT
data_file.content AS file_content,
COUNT(*) AS entry_count
FROM local.demo.entries_content_bug.entries
WHERE data_file.content != 2
GROUP BY data_file.content
ORDER BY file_content;
```
The same issue affects `all_entries`:
```sql
SELECT
data_file.content AS file_content,
COUNT(*) AS entry_count
FROM local.demo.entries_content_bug.all_entries
WHERE data_file.content != 2
GROUP BY data_file.content
ORDER BY file_content;
```
## Actual behavior
The delete manifest is pruned, and entries with:
```text
data_file.content = 1
```
are missing from the result.
The filtered query return only entries with:
```text
data_file.content = 0
```
even though position delete entries satisfy the predicate.
## Expected behavior
For:
```sql
WHERE data_file.content != 2
```
both of the following values should be returned when present:
```text
0 = FileContent.DATA
1 = FileContent.POSITION_DELETES
```
Only entries with:
```text
2 = FileContent.EQUALITY_DELETES
```
should be filtered out.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
