fhan688 opened a new pull request, #20409:
URL: https://github.com/apache/hudi/pull/20409
### Describe the issue this Pull Request addresses
When `hoodie.fail.on.timeline.archiving=false`, per-instant archival
failures are tolerated so the main write can continue. However, both timeline
archiver implementations could still delete the original active timeline files.
This could cause metadata loss:
1. One or more timeline instants fail to archive.
2. The failure is tolerated.
3. The full candidate batch is deleted from the active timeline.
4. Metadata not successfully archived is no longer available from either
timeline.
For the V2 LSM timeline, `LSMTimelineWriter` could also publish a partially
written Parquet file to the manifest. A retry could then treat the
deterministic file name as committed without validating its contents.
For V1, active instants could be deleted before the archive log writer was
closed and synced. A close or `hsync` failure could therefore occur after
deletion.
### Summary and Changelog
This PR makes timeline archival fail closed at the batch level.
- Abort the archival batch when any instant or action fails.
- Delete active timeline files only after the complete batch is durably
archived.
- Keep `hoodie.fail.on.timeline.archiving` responsible only for exception
propagation:
- `true`: propagate the archival failure.
- `false`: retain active timeline files, record the failure, and return
`0`.
- Report failed archival attempts with `archivalStatus=-1`.
#### V1 Archived Timeline
- Stop swallowing individual instant archival failures.
- Close and sync the archive log writer before deleting active timeline
files.
- Treat writer initialization and close failures as archival failures.
- Preserve close failures as suppressed exceptions when another failure
already exists.
- Clear the writer reference after closing to prevent reuse of a closed
writer.
#### V2 LSM Archived Timeline
- Stop processing the batch after an `ActiveAction` fails.
- Delete uncommitted L0 Parquet files after batch write failures.
- Do not update the manifest after a partial batch failure.
- Validate that an existing committed file contains every expected
`(instantTime, action)` pair before treating it as an idempotent retry.
- Reject legacy partial files with the expected deterministic file name but
incomplete contents.
No code was copied from another project.
### Impact
- No public API changes.
- No configuration or default-value changes.
- No archive storage format changes.
- Successful archival behavior remains unchanged.
- When archival failure is tolerated, the affected batch remains in the
active timeline for a later retry.
- Committed-file validation only runs during V2 retry or recovery when the
deterministic archive file already exists.
- A V1 retry may append duplicate archive entries if an earlier attempt
partially succeeded, but active metadata is retained and metadata loss is
prevented.
### Risk Level
`medium`
The change affects failure handling and deletion ordering in both V1 and V2
timeline archivers.
The risk is mitigated by regression coverage for:
- V1 instant metadata conversion failure.
- V1 archive writer close/`hsync` failure.
- V2 per-action callback failure.
- V2 partial batch cleanup and successful retry.
- Detection of committed but incomplete V2 archive files.
- Existing successful archival behavior.
```bash
mvn -pl hudi-client/hudi-client-common \
-DskipITs \
-DfailIfNoTests=false \
-Dtest=TestHoodieTimelineArchiver,TestArchivedTimelineV2 \
test
Test result:
- Tests run: 11
- Failures: 0
- Errors: 0
- Checkstyle violations: 0
```
### Documentation Update
None.
This PR does not add or modify any user-facing configuration. It corrects
the behavior of the existing `hoodie.fail.on.timeline.archiving` option.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
