yihua opened a new pull request, #1012:
URL: https://github.com/apache/hudi-rs/pull/1012

   ## Description
   
   closes #1009
   
   A `Table` handle now reads the metadata table as of the data snapshot it 
opened on:
   
   - The valid-instant set for a read is built from the listing the handle's 
timeline was loaded from (pending instants of every action, completed rollbacks 
and restores), not from fresh listings.
   - The metadata table is opened pinned to the snapshot: every metadata 
compaction that may have folded in an instant the snapshot cannot see is 
dropped from its timeline, so reads take the file slice from before it. 
Planning, the public metadata read APIs and table stats all read this pinned 
table.
   - When the metadata table has archived the instants that earlier slice 
needs, or its cleaner removed the slice, the handle refuses the metadata table 
and plans from a storage listing. When a pinned `files` partition read fails, 
as after a clean that runs once the table is pinned, that plan lists storage 
too. Write paths keep the latest view.
   
   One deliberate deviation from Java: the set is capped at the newest instant 
time the handle listed, so a metadata-only delta commit newer than the snapshot 
is excluded. Java loads the metadata timeline together with the data timeline 
and needs no cap; here the metadata table is opened at the first read, and an 
instant above the cap was created after the snapshot.
   
   Impact: a plan no longer lists the data timeline again for pending and 
rollback instants, which is 2 fewer LISTs per plan. A handle whose metadata 
table has nothing past its snapshot does no extra I/O; the metadata partitions 
are listed for the cleaned-slice check only when pinning drops a compaction.
   
   Known limits: a partially cleaned earlier slice is caught only when reading 
it fails, so a clean interrupted after removing the slice's logs but not its 
base reads without error and can miss the files those logs recorded. Storage 
listing cannot see a base file a later data clean deleted, so a plan that falls 
back can come back short, as on a table without a metadata table.
   
   Over half of the added lines are tests. This textually conflicts with #985 
and #1011 in the timeline loader, and needs a one-character change after #984.
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [ ] Manual tests
     - [ ] Details are described below
   
   The `snapshot_fence` tests cover a stale handle after a commit and clean, 
after a metadata compaction past its snapshot, with a metadata table archived 
or cleaned past its snapshot, and with its pinned slice cleaned after the pin 
(which fails with `No schema available for merge output` without the planning 
fallback). They also check that a compaction that ran before an unseen commit 
completed is kept, that a fresh handle reads the latest metadata, and that a 
data instant pending at open stays excluded once it completes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to