[
https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18076416#comment-18076416
]
Ivan Andika commented on HDDS-15120:
Feasibility Verdict (from AI)
Implementing bucket forks in Apache Ozone is feasible, but not as a small
extension of snapshots. A useful prototype is very feasible; a
production-ready, zero-copy, mutable, Git-like bucket fork feature is a major
OM metadata project.
The strongest path is not “make snapshots writable.” Current snapshots are
explicitly read-oriented checkpoint DBs: OmSnapshot wraps metadata reads only,
and checkpoint metadata managers are opened read-only by default in
OmSnapshot.java (line 53) and OmMetadataManagerImpl.java (line 238). Instead,
the best prototype would be a fork bucket that stores only fork-local deltas in
active OM metadata while falling back to a retained base snapshot for unchanged
keys.
Why Ozone Is A Good Fit
Ozone already has the two ingredients forks need:
Immutable data blocks and snapshot-aware retention. The docs say snapshots
duplicate metadata pointers, not data blocks, and retain blocks while
referenced by the live bucket or snapshots. See Ozone Snapshot docs.
O(1) snapshot creation is already an explicit design goal in code:
OMSnapshotCreateRequest.java (line 179) avoids key-table walks, and
OMSnapshotCreateResponse.java (line 62) writes SnapshotInfo then creates the
RocksDB checkpoint.
This aligns well with the model in HDDS-15120: fork creation should be cheap,
data should be shared until changed, and forks should isolate agent writes. It
also matches external precedent: Neon branching uses isolated copy-on-write
branches, and Tigris snapshots/forks describes isolated zero-copy bucket forks.
Main Blockers
The hard part is metadata, not data blocks.
Current snapshot chains are linear. SnapshotChainManager.java (line 39)
maintains chronological global and per-bucket chains, and rejects non-linear
additions around line 97 (line 97). Forks naturally form a DAG.
GC assumes linear ancestry. SnapshotDeletingService.java (line 147) moves
deleted entries to the next active snapshot or AOS. ReclaimableKeyFilter.java
(line 76) checks prior snapshots to decide reclaimability. Forks require
reachability/refcount semantics across multiple children, not just
“previous/next.”
Existing bucket links are not forks. Link buckets resolve requests to the
source bucket, so writes would hit the parent, not an isolated fork. See
OzoneManagerUtils.java (line 146) and ResolvedBucket.java (line 106).
Existing deleted tables cannot represent fork tombstones safely. A fork
deleting a base key should hide it from the fork without freeing base blocks.
Current deleted tables are part of physical deletion flow, so fork deletes need
separate logical tombstone metadata.
S3 API support is initially awkward. Ozone snapshot management is not available
via S3 today; the docs say snapshot creation/list/delete are managed via Ozone
RPC/CLI, though snapshot data can be read through .snapshot paths. See Known
Issues.
Recommended Design Direction
Start with a delta-overlay fork bucket:
fork create /vol/src-bucket /vol/fork-bucket --from-snapshot S creates a new
bucket with a pointer to a retained base snapshot.
Reads check fork-local metadata first, then fall back to the base snapshot.
Writes create normal fork-local keys.
Deletes of base-only keys create fork tombstones, not deletedTable entries.
Overwrites create fork-local keys plus tombstones hiding the base version.
The base snapshot is internal or refcounted so blocks remain protected while
forks exist.
This avoids multiple writable RocksDB instances and keeps OM Ratis mostly
centered on active om.db. It still requires careful changes in key
lookup/listing, delete/rename semantics, quota accounting, bucket metadata, and
cleanup.
I would avoid a first implementation that mutates checkpoint DBs directly. It
sounds tempting because snapshot checkpoints are already zero-copy, but it
pulls in multi-writable metadata stores, HA bootstrap, write routing, cache
eviction, compaction, lock semantics, and Ratis replay across multiple DBs.
That is the deep end.
Prototype Scope
A realistic first prototype:
Fork from an existing active snapshot only.
OBS bucket layout first, FSO later.
CLI/RPC only, no S3 management API initially.
Support get/list/put/delete/overwrite.
Defer rename, multipart edge cases, bucket quota precision, lifecycle,
replication tooling, and fork merge.
Keep fork base snapshots hidden from normal users but visible in admin/debug
output.
This would be enough to validate the architecture without pretending the whole
feature is done.
Bottom Line
Bucket forks are a strong fit for Ozone’s immutable-block architecture, but the
existing snapshot implementation is read-only and linear. The feature is
feasible if implemented as “new muta
