kbuci opened a new pull request, #18215:
URL: https://github.com/apache/hudi/pull/18215

   ### Summary and Changelog
   
   **Summary:** Metadata table (MDT) archival should be able to archive 
instants that are before the earliest commit to retain (in latest clean) on the 
data table, provided that there are no savepoints. This is since 
   - the metadata table anyway cannot be safely restored to a point before the 
earliest commit to retain (ECTR) on the data table
   - For datasets with infrequent ingestion writes and duration based 
compaction scheduling on MDT, there can be 1.5-2x the number of instant files 
in MDT active timeline compared to data table timeline. This is since each 
write to the data table will not only create a deltacommit on MDT, but most 
likely compaction commit as well. 
   
   **Changelog:**
   
   - **TimelineUtils**
     - `getEarliestInstantForMetadataArchival` now takes a third parameter: 
`Option<String> earliestUncleanedDataTableInstantTimeOption` (ECTR from the 
latest clean).
     - Logic uses “earliest possible restore commit” (ECTR if present and on 
timeline, else first commit), “smallest” of that and inflight, and when not 
archiving beyond savepoint, the minimum of first savepointed write, earliest 
inflight, and earliest possible restore commit.
     - Added `getEarliestRetainedCommitFromLastClean(HoodieTableMetaClient)` to 
read ECTR from the latest completed clean; throws `HoodieIOException` on read 
failure.
     - Added `findSmallestInstant(Option<HoodieInstant>, 
Option<HoodieInstant>)` and 
`getEarliestPossibleRestoreCommit(HoodieActiveTimeline, Option<String>)`.
   - **HoodieTimeline / BaseHoodieTimeline**
     - New method `findFirstSavepointedWrite()`: returns the first write commit 
that has been savepointed (used when blocking MDT archival on savepoints).
   - **TimelineArchiverV1 and TimelineArchiverV2**
     - When computing the earliest instant to retain for the metadata table, 
both archivers now pass the data table ECTR: 
`TimelineUtils.getEarliestRetainedCommitFromLastClean(dataMetaClient)` is 
supplied as the third argument to `getEarliestInstantForMetadataArchival`.
   - **Tests**
     - **TestTimelineUtils:** All `getEarliestInstantForMetadataArchival` call 
sites updated to the 3-arg signature; added cases for ECTR present, ECTR not on 
timeline, and savepoints + ECTR (earlier than savepoints and between 
savepoints).
     - **HoodieTestTable:** Added `addIncrementalClean(String instantTime, 
String earliestCommitToRetain)` for tests that need a clean with a given ECTR.
     - **TestHoodieTimelineArchiver:** Added 
`testArchivalInMetadataTableCanProceedUntilECTR` to assert MDT archival 
proceeds up to the data table ECTR.
   
   ### Impact
   
   - **User-facing / API:** No new public APIs. MDT archival may retain fewer 
instants (archive more)
   - **Performance:** Slightly less MDT timeline to scan when ECTR is present, 
due to more aggressive archival.
   
   ### Risk Level
   
   **Low.** It should be safe if instants on the data table before the ECTR 
don't have a corresponding deltacommit on MDT. This is since those instants 
must already have been committed, and cannot anyway be rolled back.
   
   ### Documentation Update
   
   None.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to