ravjot28 opened a new issue, #20584:
URL: https://github.com/apache/hudi/issues/20584

   ### Bug Description
   
   **What happened:**
   
   `restoreToSavepoint` (and `CALL rollback_to_savepoint`) fails after the data 
table rollbacks, in its metadata step, and
   leaves a pending restore on the data table, in three cases:
   
   - A. The restore deletes and rebuilds the metadata table (savepoint at or 
before its oldest compaction), and the rebuilt
     table compacts right after its bootstrap delta commits.
   - B. The metadata table was initialized after the savepoint and holds a 
later data table delta commit.
   - C. Table version 6, whenever the restore deletes and rebuilds the metadata 
table.
   
   Cases A and B (table versions 8 and later):
   
   ```
   org.apache.hudi.exception.HoodieRestoreException: Failed to restore to 
<savepoint>
     at 
org.apache.hudi.client.BaseHoodieWriteClient.restoreToSavepoint(BaseHoodieWriteClient.java)
     ...
     at 
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java)
     at 
org.apache.hudi.table.action.restore.BaseRestoreActionExecutor.finishRestore(BaseRestoreActionExecutor.java)
   Caused by: org.apache.hudi.exception.HoodieException: Unable to restore to 
instant: <savepoint>
     at 
org.apache.hudi.table.action.rollback.RestorePlanActionExecutor.execute(RestorePlanActionExecutor.java)
   Caused by: org.apache.hudi.exception.HoodieException: Unable to find 
completion time for instant: <savepoint>
   ```
   
   Case C:
   
   ```
   org.apache.hudi.exception.HoodieMetadataException: Can't restore to 
<savepoint> since there is no base file in MDT lesser than the commit to 
restore to. Please delete metadata table and retry
     at 
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java)
   ```
   
   In case C the log first shows `Deleting MDT before restore to <savepoint>`, 
so the metadata table the message asks to
   delete is the one the same restore has just rebuilt.
   
   **What you expected:**
   
   The restore completes. A metadata table rebuilt for the restore, or 
initialized after the savepoint, has nothing to
   roll back.
   
   **Steps to reproduce:**
   
   Case A:
   1. Copy-on-write table with the metadata table enabled and 
`hoodie.metadata.compact.max.delta.commits=1` (any value at
      or below the number of metadata partitions initialized at bootstrap; 
files, column stats and partition stats give 3).
   2. Insert commit C1, savepoint C1, insert commits C2 and C3.
   3. `client.restoreToSavepoint(C1)` or `CALL rollback_to_savepoint(table => 
't', instant_time => 'C1')`.
   
   Case B:
   1. Copy-on-write table with `hoodie.metadata.enable=false`. Insert commit 
C1, savepoint C1, insert commit C2.
   2. Set `hoodie.metadata.enable=true` and insert commit C3.
   3. `client.restoreToSavepoint(C1)`.
   
   Case C: case B on a table at table version 6 
(`hoodie.write.table.version=6`).
   
   **Root cause (master d5230b69e22):**
   
   - 
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:875-880`
 deletes the
     metadata table and asks for no re-initialization, but 
`initTable(operationType, instantTime, initMetadataTable)`
     ignores the flag (`BaseHoodieWriteClient.java:1500-1502`), so the metadata 
table is rebuilt, and its table services
     run 
(`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:398-399`),
 before
     the data table rollbacks.
   - 
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/RestorePlanActionExecutor.java:113-116`:
     `isMetadataTableRecreatedDuringRestore` decides from the last instant 
only. In case A it is the compaction after the
     bootstrap commits, so the table counts as not recreated and 
`RestorePlanActionExecutor.java:88-89` looks up the
     completion time of the data savepoint, which the rebuilt metadata timeline 
never held.
   - `BaseHoodieWriteClient.java:903` (`shouldDeleteMdtBeforeRestore`) deletes 
the metadata table only when the target is
     at or before its oldest compaction or before its timeline start. A 
metadata table initialized after the savepoint
     starts at `00000000000000000`, so it is kept although its bootstrap state 
includes C2 (case B); the plan then fails
     the same lookup.
   - Table version 6 initializes a metadata table at the latest data table 
instant plus a suffix (`<instant>010`), later
     than the savepoint, so the base file check in
     
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:1206-1213`
     throws for the rebuilt table (case C).
   
   The recreated-table rule came with #13691.
   
   Related, found while fixing the cases above: a metadata table rebuilt for a 
restore is built before the data table
   rollbacks, from files that include the rolled back commits, and the sync 
after the restore covers the files and column
   stats partitions only (`HoodieBackedTableMetadataWriter.java:1239-1251`; 
`MetadataPartitionType.java:425-430` keeps the
   record index on restore). With the record index enabled it keeps entries for 
keys that only the rolled back commits
   wrote. Once the cases above complete, more restores take that path, so the 
fix drops the record index partition of
   such a table during the restore, with the other index drops and before the 
metadata restore step.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1
   **Query engine:** Spark 3.5 (write client and Spark SQL procedures); the 
planning code is engine independent
   **Relevant configs:** `hoodie.metadata.enable=true`; case A needs 
`hoodie.metadata.compact.max.delta.commits` at or
   below
   the number of metadata partitions initialized at bootstrap (with the default 
of 10, ten or more partitions). Cases A and
   B reproduced on table versions 9 and 10, case C on table version 6.
   
   ### Logs and Stack Trace
   
   See "What happened" above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to