yangshangqing95 opened a new issue, #17345:
URL: https://github.com/apache/iceberg/issues/17345
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
### Issue Description
Queries with negated filters on columns except `reference_snapshot_id` of
the `all_manifests` metadata table will incorrectly return no rows.
For example, suppose the `all_manifests` metadata table contains both data
and delete manifests:
```sql
SELECT content, path, reference_snapshot_id
FROM catalog.db.table.all_manifests;
```
The result includes rows with both:
```text
content = 0 -- data manifests
content = 1 -- delete manifests
```
Filtering for data manifests works as expected:
```sql
SELECT *
FROM catalog.db.table.all_manifests
WHERE content = 0;
```
However, the following query incorrectly returns zero rows:
```sql
SELECT *
FROM catalog.db.table.all_manifests
WHERE content != 0;
```
An explicit `NOT` expression has the same problem:
```sql
SELECT *
FROM catalog.db.table.all_manifests
WHERE NOT(content = 0);
```
### Reproduction
Create a table with spark:
```sql
CREATE TABLE local.demo.t1_dv (
id BIGINT,
data STRING
)
USING iceberg
TBLPROPERTIES (
'format-version' = '3',
'write.delete.mode' = 'merge-on-read'
);
```
Append several rows and create a delete file:
```sql
INSERT INTO local.demo.t1_dv
VALUES (1, 'a'), (2, 'b'), (3, 'c');
DELETE FROM local.demo.t1_dv
WHERE id = 1;
```
Confirm that delete manifests exist:
```sql
SELECT content, path, reference_snapshot_id
FROM local.demo.t1_dv.all_manifests
ORDER BY content, path, reference_snapshot_id;
```
Then run:
```sql
SELECT content, path, reference_snapshot_id
FROM local.demo.t1_dv.all_manifests
WHERE content != 0;
```
### Expected behavior
The query should return all delete manifest rows where `content != 0`.
The following predicates should produce equivalent results:
```sql
content != 0
content <> 0
NOT(content = 0)
```
### Actual behavior
The query returns zero rows even though delete manifests are present.
### Root cause
`AllManifestsTable.SnapshotEvaluator` is a may-match evaluator used to
determine whether a snapshot may contain matching manifest rows.
Predicates on columns other than `reference_snapshot_id`, such as `content`,
return `true` because they cannot be evaluated while planning snapshot tasks
and may still match manifest rows.
However, when such a predicate is wrapped in `NOT`, the Boolean visitor
negates that result:
```text
content = 0 -> true because rows might match
NOT(content = 0) -> false
```
The `false` result causes the corresponding snapshot task to be incorrectly
pruned.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
