cshuo opened a new issue, #20443:
URL: https://github.com/apache/hudi/issues/20443
**Description**
Spark time travel queries can incorrectly return no rows because partition
pruning uses the latest partition statistics instead of statistics valid for
the requested historical snapshot.
Partition statistics can shrink after updates rewrite base files or after
compaction. A value present in a historical snapshot may therefore fall outside
the latest partition bounds. Using those bounds to prune a historical query can
exclude a partition that contains matching records.
**Example**
Consider a partitioned table with column stats and partition stats enabled:
| Snapshot | Partition `dt='p1'` | Partition `dt='p2'` |
|---|---|---|
| First commit | `(id=1, value=10)` | `(id=2, value=100)` |
| Latest commit | `(id=1, value=50)` | `(id=2, value=100)` |
After updating record `id=1`, the latest statistics for `value` are:
- Partition `dt='p1'`: `[50, 50]`
- Partition `dt='p2'`: `[100, 100]`
For COW, the update rewrites the base file. For MOR, compaction can produce
the same narrowed bounds.
A query against the first commit with `WHERE value = 10` should return
`(id=1, value=10)`. However, partition stats pruning uses the latest bounds and
skips both partitions, producing an empty result.
**Steps to reproduce**
1. Create a partitioned COW or MOR table with column stats and partition
stats enabled for `value`.
2. Insert `(id=1, value=10, version=1000, dt='p1')` and `(id=2, value=100,
version=1000, dt='p2')`. Save the first commit instant.
3. Upsert `(id=1, value=50, version=1001, dt='p1')`. For MOR, enable inline
compaction so the latest file statistics exclude the old value.
4. Read the first commit with data skipping enabled:
```scala
spark.read.format("hudi")
.option("as.of.instant", firstCommit)
.option("hoodie.enable.data.skipping", "true")
.load(basePath)
.filter("value = 10")
.select("id", "value", "version", "dt")
.show()
```
**Expected behavior**
The query returns the historical record:
```text
id=1, value=10, version=1000, dt=p1
```
**Actual behavior**
The query returns no rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
