yihua opened a new pull request, #985:
URL: https://github.com/apache/hudi-rs/pull/985

   ## Description
   
   On table version 6 a log file is named on its file slice's base instant, so 
writes made while a compaction is pending carry the compaction's instant, and 
the listing dropped them because that instant has not completed. The timeline 
now also loads compaction instants from its existing listing and keeps the 
pending ones in their own set, apart from the log-block gate's inputs and the 
archival boundary. A log file whose name instant is a pending compaction is 
admitted and merged into the slice before the compaction, as Java's file system 
view does (`filterCompletedAndCompactionInstants`, `fetchMergedFileSlice`). 
Base files still need a completed commit, and failed writes among the admitted 
log files are still excluded per block by file group reader version 2.
   
   Closes #916
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [ ] Manual tests
     - [ ] Details are described below
   
   A new Spark-written v6 MOR fixture (`table_pending_compaction_v6`, with gold 
data) has two delta commits written while a scheduled compaction is pending. 
Snapshot, time-travel and incremental reads on both reader versions match 
Spark; on main they return the pre-compaction values. A doctored failed write 
against the pending compaction stays excluded, and a timeline unit test pins 
the pending-compaction split.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to