yihua opened a new pull request, #1011:
URL: https://github.com/apache/hudi-rs/pull/1011

   ## Description
   
   closes #1008
   
   Opening a table now lists the layout-2 active timeline directory 
(`.hoodie/{hoodie.timeline.path}`) concurrently with the `hoodie.properties` 
read. The timeline loader reuses that listing only when the resolved configs 
select the same directory on the same base URL, and the listing holds a file 
the store stamped strictly after `hoodie.properties`. A writer changes the 
timeline before it rewrites the properties (a layout upgrade migrates the 
timeline first), so only such a file shows the listing was taken after the 
properties that were read. Both times come from the store, so the reader's 
clock does not enter. Equal times, an empty or failed listing, or a properties 
time at the Unix epoch (a store that keeps no last-modified time) list the 
timeline again, and a properties failure is still returned as before.
   
   Impact: opening a table on S3 whose latest timeline file is newer than 
`hoodie.properties` (TPC-H SF100 `orders` and `region`) goes from 2 sequential 
requests (~96-140 ms) to 1 step with the GET and LIST in flight together 
(~50-55 ms), one round trip saved per table open and per metadata table open. 
When the timeline has no file newer than the properties (a table copied with 
uniform timestamps, or a properties rewrite after the last commit), the listing 
is not reused and the open lists again: one extra LIST, with the same latency 
as before. On a layout-1 table the early listing is one unused LIST.
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [x] Manual tests
     - [x] Details are described below
   
   The `table::builder` tests open tables through a test store that records 
when each GET and LIST starts and ends. They check that the timeline LIST 
overlaps the properties GET and is not repeated, and that the timeline is 
listed again for a layout-1 table, a failed listing, an overridden timeline 
path, a timeline no newer than the properties, and properties stamped at the 
Unix epoch. A replayed mid-upgrade open (timeline listed before the migration 
finished, upgraded properties read after) lists again with the store clock in 
step with the host and a day behind it. Each opened timeline is compared with 
one loaded without the early listing. The S3 latencies above come from request 
traces of table opens.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to