ravjot28 opened a new pull request, #20594:
URL: https://github.com/apache/hudi/pull/20594

   ### Describe the issue this Pull Request addresses
   
   Closes #20584
   
   A restore to a savepoint fails in its metadata step, and leaves a pending 
restore on the data table, when the metadata
   table does not hold the savepoint: (A) the restore rebuilt the metadata 
table and the rebuilt table compacted right
   after its bootstrap commits; (B) the metadata table was initialized after 
the savepoint; (C) table version 6, whenever
   the restore rebuilds the metadata table. A and B fail with `Unable to find 
completion time for instant`, C with `Can't
   restore to ... since there is no base file in MDT lesser than the commit to 
restore to`.
   
   ### Summary and Changelog
   
   A restore to a savepoint completes in the three cases.
   
   - `HoodieTableMetadataUtil#hasOnlyInitializationDeltaCommits`: true when 
every delta commit on a metadata table's
     active timeline initializes a metadata partition (`SOLO_COMMIT_TIMESTAMP` 
prefix from table version 8; before it,
     the latest data table instant plus a suffix of `010` or above). Table 
service instants do not change the answer
     (compaction and clean; a completed log compaction on the metadata table, 
off by default, still counts as a data
     table delta commit from table version 8).
   - `RestorePlanActionExecutor` uses it and plans no rollbacks for such a 
table; it decided from the last instant only,
     which is the compaction in case A.
   - `HoodieBackedTableMetadataWriter#update(HoodieRestoreMetadata, String)` 
skips its base file check for such a table
     (case C) and reloads the metadata timeline only when the check fails.
   - `BaseHoodieWriteClient#shouldDeleteMdtBeforeRestore` also deletes the 
metadata table when the target is a completed
     data table instant that the metadata timeline does not hold and the 
metadata table holds delta commits of data table
     instants (case B), so it is rebuilt. This additional deletion rule applies 
only to completed data table instants.
     Existing deletion rules for targets at or before the oldest metadata 
compaction, or before the metadata timeline
     start, are unchanged. A metadata table that holds only initialization 
delta commits is restored in place, as before.
   - Record index: a metadata table that holds only initialization delta 
commits was built from files that include the
     commits the restore rolls back, and the sync after the restore covers the 
files and column stats partitions only.
     Its record index would map keys of rolled back commits to removed files. 
`BaseActionExecutor#writeTableMetadata`
     now drops the record index partition of such a table when the restore plan 
rolls back at least one instant, right
     after `dropIndexOnRestore()` drops the other index partitions, with a 
backup and on the same table config. The plan
     decides, not the instants the current attempt rolls back, so a restore 
retried after an earlier attempt finished
     its rollbacks drops it too. The drop comes before the metadata restore and 
its sync commit, so a failure after it
     leaves no stale record index either; the next writer with the record index 
enabled rebuilds it from the restored
     data table. This also covers the existing delete-and-rebuild path (target 
at or before the oldest metadata
     compaction).
   - New tests: 
`TestSavepointRestoreCopyOnWrite#testRestoreWithMetadataCompactionAfterEveryDeltaCommit`
 (table versions
     6, 9, 10), `#testRestoreToSavepointOlderThanMetadataTable` (6, 9, 10, with 
a commit after the restore),
     `#testRestoreTargetWithoutInstantKeepsMetadataTable`; in 
`hudi-spark-datasource/hudi-spark`, because the record
     index initializes through the Spark adapter, a new class 
`TestSavepointRestoreRecordIndex` with
     
`#testRetriedRestoreDropsTheRecordIndexAfterAnEarlierAttemptRolledBackTheData` 
(the retry rolls back nothing and
     still drops the record index) and
     `#testRestoreLeavesNoRecordIndexEntryOfRolledBackCommits` (record index as 
the write index; metadata table compacted
     after every delta commit, enabled after the savepoint, or initialized just 
before the restore and restored in place:
     the table config lists neither the record index nor the partition stats 
the restore dropped, the record index holds
     no key of a rolled back commit, and an upsert of such a key after the 
restore is written as a new record), and
     `TestHoodieTableMetadataUtil#testHasOnlyInitializationDeltaCommits`.
   
   `initTable(operationType, instantTime, initMetadataTable)` ignores its flag, 
which is why the metadata table is rebuilt,
   and table services run on it, before the restore plans its rollbacks. This 
change leaves that alone: honouring the flag
   moves the rebuild to the next writer, a wider behaviour change than this fix 
needs.
   
   ### Impact
   
   No public API or config change. Restores that failed with the errors above 
complete. A restore (savepoint restore, and
   the `restore_to_instant` procedure) rebuilds the metadata table in one more 
case: the target is a completed data table
   instant that the metadata timeline does not hold, while the metadata table 
holds delta commits of later data table
   instants. After a restore of a metadata table that holds only initialization 
delta commits, its record index partition
   is deleted and rebuilt by the next writer: a full read of the record keys of 
the restored table.
   
   Related open PRs on the same files: #18572 (tests for 
`deleteMdtIfNecessaryBeforeRestore`) and #20310 (edits
   `restoreToSavepoint` a few lines above this change); whichever lands second 
needs a small rebase.
   
   **Backport:** please include this fix in the next 1.2.x patch release; the 
defect is present in 1.2.1. The commit does
   not cherry-pick cleanly onto `release-1.2.1`: `BaseActionExecutor` conflicts 
in its imports, and
   `TestSavepointRestoreCopyOnWrite` with test changes that are only on master. 
I can open an adapted change against
   `branch-1.2.x`.
   
   ### Risk Level
   
   medium. The new predicate applies only to metadata tables; a metadata table 
that holds a delta commit of a data table
   instant is planned and checked as before. For a completed data table instant 
the added delete rule fires where the
   in-place metadata restore failed before. The record index deletion adds a 
rebuild after restores that, until now,
   completed with a record index that could point at removed files.
   
   Tests, run on master with JDK 11 and passing checkstyle and scalastyle 
checks, with
   `mvn test -pl hudi-client/hudi-spark-client,hudi-spark-datasource/hudi-spark 
-am -Dtest=...` and the classes listed
   below:
   - Without the main code change (`TestHoodieTableMetadataUtil`, the unit test 
of the new helper, left out): 5 of the 14
     cases of `TestSavepointRestoreCopyOnWrite` fail with `Failed to restore to 
<savepoint>` (completion time not found,
     table versions 9 and 10; the missing base file, table version 6). With 
only `BaseActionExecutor` reverted to master,
     the three cases of 
`TestSavepointRestoreRecordIndex#testRestoreLeavesNoRecordIndexEntryOfRolledBackCommits`
 fail on
     "the record index is dropped"; with the drop decided from the instants of 
the current attempt, the retried restore
     case fails on "the retried restore drops the record index".
   - With the change: `TestSavepointRestoreCopyOnWrite` 14/14, 
`TestSavepointRestoreRecordIndex` 4/4,
     `TestSavepointRestoreMergeOnRead` 18/18, `TestHoodieTableMetadataUtil` 
41/41 pass; and the existing tests of the
     touched methods: `TestHoodieBackedMetadata` 94 (2 skipped), 
`TestJavaHoodieBackedMetadata` 68/68,
     `TestClientRollback` 26/26, 
`TestSavepointRestoreCopyOnWriteWithTestFormat` 5/5, ScalaTest 
`TestRestoreProcedure`,
     `TestSavepointsProcedure` and `TestGlobalRecordLevelIndex` 17/17.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to