RyanLin5967 opened a new issue, #921:
URL: https://github.com/apache/hudi-rs/issues/921
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Description of the bug
Currently, each read stores its stats-pruned file groups in the handle's
shared `partition_to_file_groups` map and then collects slices from the whole
map, so a concurrent filtered read replaces another read's file lists with its
pruned ones. A handle per thread returns all 400 rows every time.
### Steps To Reproduce
```python
import tempfile, threading
from pyspark.sql import SparkSession
from hudi import HudiTableBuilder, HudiReadOptions
spark = (SparkSession.builder
.config("spark.jars.packages",
"org.apache.hudi:hudi-spark4.0-bundle_2.13:1.2.1")
.config("spark.serializer",
"org.apache.spark.serializer.KryoSerializer").getOrCreate())
path = tempfile.mkdtemp()
spark.createDataFrame([(i, i % 40) for i in range(400)], "id INT, p
INT").write.format("hudi") \
.options(**{"hoodie.table.name": "t",
"hoodie.datasource.write.recordkey.field": "id",
"hoodie.datasource.write.partitionpath.field":
"p"}).save(path)
t = HudiTableBuilder.from_base_uri(path).build()
def count(filters, out):
out.append(sum(b.num_rows for b in
t.read(HudiReadOptions().with_filters(filters))))
full, one = [], []
for _ in range(20):
ts = [threading.Thread(target=count, args=([("id", ">=", "0")], full)),
threading.Thread(target=count, args=([("id", "=", "7")], one))]
[x.start() for x in ts]; [x.join() for x in ts]
print(all(n == 400 for n in full), set(one))
# False {1}, should be True {1}
# the first filter matches every row, and run alone it returns 400
```
### Expected behavior
The first read returns all 400 rows whatever another read on the same handle
filters on.
### Screenshots / Logs
_No response_
### Software information
- Operating system: macOS 26.7.1, arm64 (Darwin 25.6.0)
- Rust version: none, the PyPI wheel on Python 3.12.16
- Project version: hudi 0.5.0, and main (its CI wheel, 0.6.0.dev0)
- Writer: Hudi 1.2.1 (`hudi-spark4.0-bundle_2.13`) on pyspark 4.0.5, Java
21.0.12.1
### Additional context
An unfiltered `read()` next to the filtered one comes back short too, 10
times in 100 on main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
