yihua opened a new pull request, #764:
URL: https://github.com/apache/hudi-rs/pull/764
## Description
`HudiDataSource` now caches resolved file slices per (pushdown filters,
read-optimized) pair, so repeated `scan()` calls against the same registered
provider resolve the table's file-system view and metadata table once instead
of on every query. This is sound because the provider is a construction-time
snapshot of the timeline (later commits are invisible to it either way), so
identical scans always resolve identical slices. Resolution is serialized per
provider by holding the cache's async lock across the listing, which also makes
each cached entry the product of a single isolated file-system-view pass. Part
of the query analysis work in #713.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
New test pins that an identical repeated scan reuses the cached entry and
plans identically, that a different pushdown filter set gets its own entry, and
(via a seeded sentinel entry) that the scan actually plans from the cached
slices rather than re-resolving. The full hudi-datafusion suite passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
