cshuo opened a new issue, #20668:
URL: https://github.com/apache/hudi/issues/20668
### Description
Spark snapshot queries on a Merge-on-Read table can return incomplete
results when metadata bloom filter data skipping is enabled.
Metadata bloom filters cover base files only. However,
`BloomFiltersIndexSupport` returns candidate base filenames, and
`HoodieFileIndex` uses those candidates to decide whether to retain an entire
file slice. This can incorrectly exclude:
- A base+log slice when the queried record key exists only in its logs and
the base bloom returns negative.
- A log-only slice, which has no base filename to contribute to the
candidate set.
The records remain readable when data skipping is disabled.
### Steps to reproduce
Create a MOR table with:
```properties
hoodie.datasource.write.recordkey.field=_row_key
hoodie.datasource.write.partitionpath.field=partition
hoodie.metadata.enable=true
hoodie.metadata.index.bloom.filter.enable=true
hoodie.metadata.index.bloom.filter.column.list=_row_key
hoodie.compact.inline=false
hoodie.compact.schedule.inline=false
```
Disable other query indexes to isolate metadata bloom pruning, including
bucket query pruning, record-level indexes, and column/partition statistics.
Perform these writes without compaction:
1. Use a single-bucket writer and `insert_overwrite` to write key `a` into
partition `base-and-log`, and key `c` into partition `base-only`.
2. Upsert a new key `b` into partition `base-and-log`. The existing bucket
routes it into the existing file group's log.
3. Use the `INMEMORY` writer index and `insert` to write key `d` into a new
partition `log-only`, creating a slice without a base file.
Verify the resulting layout:
| Partition | Base records | Log records |
|---|---|---|
| `base-and-log` | `a` | `b` |
| `base-only` | `c` | — |
| `log-only` | — | `d` |
Both base-file blooms must be present and return negative for `b` and `d`.
Run snapshot queries with data skipping enabled and disabled:
```scala
def query(keys: Seq[String], skipping: Boolean): DataFrame = {
spark.read.format("hudi")
.option("hoodie.datasource.query.type", "snapshot")
.option("hoodie.enable.data.skipping", skipping.toString)
.option("hoodie.bucket.query.index", "false")
.load(basePath)
.filter(col("_row_key").isin(keys: _*))
.select("_row_key")
}
```
### Expected and actual results
| Predicate | Expected / skipping disabled | Skipping enabled |
|---|---|---|
| `_row_key = 'b'` | `b` | Empty |
| `_row_key = 'd'` | `d` | Empty |
| `_row_key IN ('b', 'd')` | `b, d` | Empty |
| `_row_key = 'c'` | `c` | `c` |
Read-optimized queries correctly exclude `b` and `d`, since those queries
only read base files.
### Root cause
`BloomFiltersIndexer` skips delta write stats and non-base files, so the
metadata bloom filters do not describe keys newly inserted into logs.
`BloomFiltersIndexSupport.computeCandidateFileNames` nevertheless derives
candidates only from base files. `HoodieFileIndex.filterFileSlices` then drops
any slice whose base/log filenames do not intersect those candidates. A
negative base bloom therefore becomes an incorrect negative result for the
whole slice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]