[
https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18076715#comment-18076715
]
Ivan Andika commented on HDDS-15120:
------------------------------------
FYI I created a quick prototype: https://github.com/ivandika3/ozone/pull/3
The initial idea is quite straightforward. We setup a new "fork" bucket with a
read through to the original snapshot. Any new keys will be created in this
"fork" bucket in keyTable (this can be visible to everyone).
However, each getKeyInfo will need to check the following
* If the key exists in a tombstone, it means that the key has been deleted only
in this fork, we return KEY_NOT_FOUND
* Return the fork bucket keyTable entry if it exists
* Check whether the key exists in the original snapshot key, return it if it
exists, otherwise return KEY_NOT_FOUND.
> Support bucket forks / branches for agentic workload
> ----------------------------------------------------
>
> Key: HDDS-15120
> URL: https://issues.apache.org/jira/browse/HDDS-15120
> Project: Apache Ozone
> Issue Type: New Feature
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Currently, Ozone supports bucket snapshot which creates a read-only immutable
> state of the entire bucket for use cases such as backup, replication,
> compliance, etc. This is achieved using the RocksDB checkpoint feature which
> tracks the current SST files at that point.
> With the recent rise agentic workloads, there is a need for storage systems
> to implement forking / branching to cater for multi-agents workload. Unlike
> snapshot, forks can be mutated. The idea of forking and branch is similar to
> Git branch / worktrees where a new "branch" is created based on the base
> directory. Multiple agents can fork the same base file system in parallel and
> mutate these forks without affecting each other. These forks should also have
> zero-copy, similar to snapshot (which should only require O(1) time to
> create). Additionally, these forks lifetime can varies (it can be retained
> for a long time or discarded quite quickly).
> Example systems
> * NeonDB branching: https://neon.com/docs/introduction/branching
> * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/
> (please see the related blogs on the implementations of forks).
> Ozone can consider supporting this feature. Since more systems implement
> storage compute separation architecture on object storage, the compute /
> caching layer can rely on Ozone as the backing store for agentic workloads
> since Ozone supports snapshot and forking (they don't need to implement
> snapshot and forking or need to write complicated logic to store their forks
> state). Ozone can then position itself as the open-source object store /
> distributed file system for agentic workloads.
> This ticket acts as a way to start a discussion in the community on this
> direction. We can start thinking about this (and probably try to start
> prototyping some ideas). This might require a radical change of Ozone Manager
> design (e.g. might need to introduce versioning, reference counting,
> copy-on-write, log subsystems, OM deletions semantic change, etc).
> Out of scope
> * Merge Ozone fork back to the main bucket
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
