yihua opened a new pull request, #1011:
URL: https://github.com/apache/hudi-rs/pull/1011
## Description
closes #1008
Opening a table now lists the layout-2 active timeline directory
(`.hoodie/{hoodie.timeline.path}`) concurrently with the `hoodie.properties`
read. The timeline loader reuses that listing only when the resolved configs
select the same directory on the same base URL, and the listing holds a file
the store stamped strictly after `hoodie.properties`. A writer changes the
timeline before it rewrites the properties (a layout upgrade migrates the
timeline first), so only such a file shows the listing was taken after the
properties that were read. Both times come from the store, so the reader's
clock does not enter. Equal times, an empty or failed listing, or a properties
time at the Unix epoch (a store that keeps no last-modified time) list the
timeline again, and a properties failure is still returned as before.
Impact: opening a table on S3 whose latest timeline file is newer than
`hoodie.properties` (TPC-H SF100 `orders` and `region`) goes from 2 sequential
requests (~96-140 ms) to 1 step with the GET and LIST in flight together
(~50-55 ms), one round trip saved per table open and per metadata table open.
When the timeline has no file newer than the properties (a table copied with
uniform timestamps, or a properties rewrite after the last commit), the listing
is not reused and the open lists again: one extra LIST, with the same latency
as before. On a layout-1 table the early listing is one unused LIST.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
The `table::builder` tests open tables through a test store that records
when each GET and LIST starts and ends. They check that the timeline LIST
overlaps the properties GET and is not repeated, and that the timeline is
listed again for a layout-1 table, a failed listing, an overridden timeline
path, a timeline no newer than the properties, and properties stamped at the
Unix epoch. A replayed mid-upgrade open (timeline listed before the migration
finished, upgraded properties read after) lists again with the store clock in
step with the host and a day behind it. Each opened timeline is compared with
one loaded without the early listing. The S3 latencies above come from request
traces of table opens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]