voonhous opened a new pull request, #18359:
URL: https://github.com/apache/hudi/pull/18359
### Describe the issue this Pull Request addresses
RFC-100 Part 2: external blob cleanup design (#18111). When the cleaner
expires file slices, the out-of-line blobs they reference can become orphaned;
this PR adds the problem statement and the cleanup design.
The active design (v3) tracks blob references at **file granularity** in a
new MDT partition, `blob_refs`, with one entry per `(external_path, partition,
file)` written by the writer in the same metadata delta commit as the FILES
entry. The registry is **append-only**: a blob is live while some entry names a
file still present in FILES, both read at one snapshot. The clean never writes
to the registry; stale entries are dropped by a filter inside MDT compaction.
No blob paths are stored in commit metadata; a commit carries one boolean,
`blobRefsWritten`.
```mermaid
flowchart LR
subgraph W["Every commit"]
WS["Distinct blob paths per<br/>written file (WriteStatus)"] -->
REG["MDT blob_refs entries,<br/>same delta commit as FILES"]
end
subgraph P["Clean plan"]
A["Unlocked: enumerate expiring<br/>files; registry join FILES @
T0"] --> B["Locked: any commit since T0<br/>with blobRefsWritten? retry"] -->
S["Plan sidecar:<br/>verified delete list"]
end
subgraph E["Clean execute"]
D["Delete slices,<br/>then blobs (idempotent)"]
end
V["Writer preCommit veto<br/>against pending and<br/>recently completed
plans"]
W --> P --> E
V -.-> S
```
Verification is one routed prefix scan per candidate, O(candidates),
independent of table size. Write-path cost is O(distinct paths per written
file). The protocol is model-checked in `rfc/rfc-100/model/` (TLA+ and Python):
the design passes; dropping the veto, the locked check, the completed-clean
veto, or reading completed-only FILES in GC each yields an R1 counterexample.
### Summary and Changelog
- `rfc-100-blob-cleaner-problem.md`: constraints C1-C11, requirements R1-R9,
failure-mode examples. Solution-free.
- `rfc-100-blob-cleaner-design-v3.md`: the active design, with two SVG
figures (architecture v2 vs v3, six sequence diagrams) and the model under
`model/`.
- `rfc-100-blob-cleaner-design-v2.md` and `rfc-100-blob-cleaner-design.md`:
superseded designs, headlined with the reasons; retained for history.
<details>
<summary>Why v2 was superseded</summary>
- v2 carried the registry source on `HoodieWriteStat`, which is serialized
into `HoodieCommitMetadata`. That stores blob references in the commit file,
which was ruled out in review and compounds C10. v3 carries the per-file path
sets on the in-memory `WriteStatus`.
- v2 tombstoned registry entries under the clean lock, O(rows) for
row-distinct blobs, serializing writers (R4). v3's FILES-membership liveness
needs no tombstones.
- v2's plan-time snapshot used the table's cached metadata reader, which the
file-system view can reset between the registry and FILES reads. v3 uses a
dedicated reader and adds the locked `blobRefsWritten` check.
- Per-file manifests (a log-block format change), the hygiene sidecar, the
reconciliation sweep and the 24h pending-deletes queue added surface without
closing any gap the model found. The queue stays as an optional later phase.
Carried forward from v2: the `blob_refs` partition with routed file groups,
the plan sidecar as the durable delete list, the two-step verify, the writer
`preCommit` veto, defer-over-annul.
</details>
<details>
<summary>Why v1 (SI+RLI) was abandoned</summary>
A secondary index makes `insert_overwrite`/`delete_partition` throw
table-wide; snapshot-semantics SI reads cannot implement the retained-slice
liveness rule (dangling refs inside the retention window); and its
`cleaned_fg_ids` heuristic deletes blobs still referenced by partially-cleaned
file groups. Details in the v2 doc's Motivation (M1-M3).
</details>
### Impact
None, RFC design.
### Risk Level
None, RFC design.
### Documentation Update
None, RFC design.
### Contributor's checklist
- [X] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [X] Enough context is provided in the sections above
- [X] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
