ravjot28 opened a new pull request, #20594:
URL: https://github.com/apache/hudi/pull/20594
### Describe the issue this Pull Request addresses
Closes #20584
A restore to a savepoint fails in its metadata step, and leaves a pending
restore on the data table, when the metadata
table does not hold the savepoint: (A) the restore rebuilt the metadata
table and the rebuilt table compacted right
after its bootstrap commits; (B) the metadata table was initialized after
the savepoint; (C) table version 6, whenever
the restore rebuilds the metadata table. A and B fail with `Unable to find
completion time for instant`, C with `Can't
restore to ... since there is no base file in MDT lesser than the commit to
restore to`.
### Summary and Changelog
A restore to a savepoint completes in the three cases.
- `HoodieTableMetadataUtil#hasOnlyInitializationDeltaCommits`: true when
every delta commit on a metadata table's
active timeline initializes a metadata partition (`SOLO_COMMIT_TIMESTAMP`
prefix from table version 8; before it,
the latest data table instant plus a suffix of `010` or above). Table
service instants do not change the answer
(compaction and clean; a completed log compaction on the metadata table,
off by default, still counts as a data
table delta commit from table version 8).
- `RestorePlanActionExecutor` uses it and plans no rollbacks for such a
table; it decided from the last instant only,
which is the compaction in case A.
- `HoodieBackedTableMetadataWriter#update(HoodieRestoreMetadata, String)`
skips its base file check for such a table
(case C) and reloads the metadata timeline only when the check fails.
- `BaseHoodieWriteClient#shouldDeleteMdtBeforeRestore` also deletes the
metadata table when the target is a completed
data table instant that the metadata timeline does not hold and the
metadata table holds delta commits of data table
instants (case B), so it is rebuilt. This additional deletion rule applies
only to completed data table instants.
Existing deletion rules for targets at or before the oldest metadata
compaction, or before the metadata timeline
start, are unchanged. A metadata table that holds only initialization
delta commits is restored in place, as before.
- Record index: a metadata table that holds only initialization delta
commits was built from files that include the
commits the restore rolls back, and the sync after the restore covers the
files and column stats partitions only.
Its record index would map keys of rolled back commits to removed files.
`BaseActionExecutor#writeTableMetadata`
now drops the record index partition of such a table when the restore plan
rolls back at least one instant, right
after `dropIndexOnRestore()` drops the other index partitions, with a
backup and on the same table config. The plan
decides, not the instants the current attempt rolls back, so a restore
retried after an earlier attempt finished
its rollbacks drops it too. The drop comes before the metadata restore and
its sync commit, so a failure after it
leaves no stale record index either; the next writer with the record index
enabled rebuilds it from the restored
data table. This also covers the existing delete-and-rebuild path (target
at or before the oldest metadata
compaction).
- New tests:
`TestSavepointRestoreCopyOnWrite#testRestoreWithMetadataCompactionAfterEveryDeltaCommit`
(table versions
6, 9, 10), `#testRestoreToSavepointOlderThanMetadataTable` (6, 9, 10, with
a commit after the restore),
`#testRestoreTargetWithoutInstantKeepsMetadataTable`; in
`hudi-spark-datasource/hudi-spark`, because the record
index initializes through the Spark adapter, a new class
`TestSavepointRestoreRecordIndex` with
`#testRetriedRestoreDropsTheRecordIndexAfterAnEarlierAttemptRolledBackTheData`
(the retry rolls back nothing and
still drops the record index) and
`#testRestoreLeavesNoRecordIndexEntryOfRolledBackCommits` (record index as
the write index; metadata table compacted
after every delta commit, enabled after the savepoint, or initialized just
before the restore and restored in place:
the table config lists neither the record index nor the partition stats
the restore dropped, the record index holds
no key of a rolled back commit, and an upsert of such a key after the
restore is written as a new record), and
`TestHoodieTableMetadataUtil#testHasOnlyInitializationDeltaCommits`.
`initTable(operationType, instantTime, initMetadataTable)` ignores its flag,
which is why the metadata table is rebuilt,
and table services run on it, before the restore plans its rollbacks. This
change leaves that alone: honouring the flag
moves the rebuild to the next writer, a wider behaviour change than this fix
needs.
### Impact
No public API or config change. Restores that failed with the errors above
complete. A restore (savepoint restore, and
the `restore_to_instant` procedure) rebuilds the metadata table in one more
case: the target is a completed data table
instant that the metadata timeline does not hold, while the metadata table
holds delta commits of later data table
instants. After a restore of a metadata table that holds only initialization
delta commits, its record index partition
is deleted and rebuilt by the next writer: a full read of the record keys of
the restored table.
Related open PRs on the same files: #18572 (tests for
`deleteMdtIfNecessaryBeforeRestore`) and #20310 (edits
`restoreToSavepoint` a few lines above this change); whichever lands second
needs a small rebase.
**Backport:** please include this fix in the next 1.2.x patch release; the
defect is present in 1.2.1. The commit does
not cherry-pick cleanly onto `release-1.2.1`: `BaseActionExecutor` conflicts
in its imports, and
`TestSavepointRestoreCopyOnWrite` with test changes that are only on master.
I can open an adapted change against
`branch-1.2.x`.
### Risk Level
medium. The new predicate applies only to metadata tables; a metadata table
that holds a delta commit of a data table
instant is planned and checked as before. For a completed data table instant
the added delete rule fires where the
in-place metadata restore failed before. The record index deletion adds a
rebuild after restores that, until now,
completed with a record index that could point at removed files.
Tests, run on master with JDK 11 and passing checkstyle and scalastyle
checks, with
`mvn test -pl hudi-client/hudi-spark-client,hudi-spark-datasource/hudi-spark
-am -Dtest=...` and the classes listed
below:
- Without the main code change (`TestHoodieTableMetadataUtil`, the unit test
of the new helper, left out): 5 of the 14
cases of `TestSavepointRestoreCopyOnWrite` fail with `Failed to restore to
<savepoint>` (completion time not found,
table versions 9 and 10; the missing base file, table version 6). With
only `BaseActionExecutor` reverted to master,
the three cases of
`TestSavepointRestoreRecordIndex#testRestoreLeavesNoRecordIndexEntryOfRolledBackCommits`
fail on
"the record index is dropped"; with the drop decided from the instants of
the current attempt, the retried restore
case fails on "the retried restore drops the record index".
- With the change: `TestSavepointRestoreCopyOnWrite` 14/14,
`TestSavepointRestoreRecordIndex` 4/4,
`TestSavepointRestoreMergeOnRead` 18/18, `TestHoodieTableMetadataUtil`
41/41 pass; and the existing tests of the
touched methods: `TestHoodieBackedMetadata` 94 (2 skipped),
`TestJavaHoodieBackedMetadata` 68/68,
`TestClientRollback` 26/26,
`TestSavepointRestoreCopyOnWriteWithTestFormat` 5/5, ScalaTest
`TestRestoreProcedure`,
`TestSavepointsProcedure` and `TestGlobalRecordLevelIndex` 17/17.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]