RyanLin5967 opened a new issue, #913:
URL: https://github.com/apache/hudi-rs/issues/913
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Description of the bug
Currently, an as-of read doesn't check the latest completed clean's
`earliestCommitToRetain`, so reading a commit the cleaner has already cleaned
skips each file group whose base file was deleted and returns the rest with no
error. With `hoodie.clean.automatic=false` the same read returns both rows.
### Steps To Reproduce
```python
import tempfile
from pyspark.sql import SparkSession
from hudi import HudiTableBuilder, HudiReadOptions
spark = (SparkSession.builder
.config("spark.jars.packages",
"org.apache.hudi:hudi-spark4.0-bundle_2.13:1.2.1")
.config("spark.serializer",
"org.apache.spark.serializer.KryoSerializer").getOrCreate())
path = tempfile.mkdtemp()
opts = {"hoodie.table.name": "t", "hoodie.datasource.write.recordkey.field":
"id",
"hoodie.datasource.write.partitionpath.field": "p",
"hoodie.clean.commits.retained": "1"}
for rows in ([("a", 1, "p1"), ("b", 2, "p2")], [("a", 3, "p1")], [("a", 4,
"p1")]):
spark.createDataFrame(rows, "id STRING, v LONG, p
STRING").write.format("hudi") \
.options(**opts).mode("append").save(path)
t = HudiTableBuilder.from_base_uri(path).build()
c1 = t.get_timeline().get_completed_commits()[0].timestamp
batches = t.read(HudiReadOptions().with_as_of_timestamp(c1))
print(sorted((r["id"], r["v"]) for b in batches for r in b.to_pylist()))
# [('b', 2)], should be an error
# the clean after the third write deleted a's base file from c1, and c1 is
still listed as a completed commit
spark.read.format("hudi").option("as.of.instant", c1).load(path).collect()
# IllegalArgumentException: Cleaner cleaned up the timestamp of interest.
Please ensure sufficient ...
# so Hudi's Spark reader refuses the same read
```
### Expected behavior
The read as of `c1` fails the way Spark's does, since the [tech
spec](https://hudi.apache.org/learn/tech-specs/#cleaning) says "Snapshot
reconstruction on a commit instant which has been cleaned is not possible."
### Screenshots / Logs
_No response_
### Software information
- Operating system: macOS 26.7.1, arm64 (Darwin 25.6.0)
- Rust version: none, the PyPI wheel on Python 3.12.16
- Project version: hudi 0.5.0, and main (its CI wheel, 0.6.0.dev0)
- Writer: Hudi 1.2.1 (`hudi-spark4.0-bundle_2.13`) on pyspark 4.0.5, Java
21.0.12.1
### Additional context
With the cleaner's defaults and each commit updating `a` in an unpartitioned
table, reading the first commit returns `[]` from the 12th commit on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
