[
https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-15120:
-------------------------------
Description:
Currently, Ozone supports bucket snapshot which creates a read-only immutable
state of the entire bucket for use cases such as backup, replication,
compliance, etc.
However, in the recent rise agentic workloads, there is a need for systems to
implement forking / branching to cater for multi-agents workload. Unlike
snapshot, forks can be mutated. The idea of forking and branch is similar to
Git branch / worktrees where a new "branch" is created based on the base
directory. Multiple agents can fork the same base file system in parallel and
mutate these forks without affecting each other. These forks should also have
zero-copy, similar to snapshot. Additionally, these forks lifetime can varies
(it can be retained for a long time or discarded quite quickly).
Example systems
* NeonDB branching: https://neon.com/docs/introduction/branching
* Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/
(please see the related blogs on the implementations for the idea).
Ozone can consider supporting this feature. With the rise of the storage
compute separation architecture on databases (OLAP or OLTP), the compute /
caching layer can rely on Ozone as the backing store for agentic workloads
since Ozone supports snapshot and forking. Ozone then can position itself as
the object store for agentic workloads.
This ticket acts as a way to start a discussion in the community on this
direction.
was:
Currently, Ozone supports bucket snapshot which creates a read-only immutable
state of the entire bucket for use cases such as backup, replication,
compliance, etc.
However, in the recent rise agentic workloads, there is a need for systems to
implement forking / branching to cater for multi-agents workload. The idea of
forking and branch is similar to Git branch / worktrees where a new "branch" is
created based on the base directory. Multiple agents can fork the same base
file system in parallel and mutate these forks without affecting each other.
These forks should also have zero-copy, similar to snapshot. Additionally,
these forks lifetime can varies (it can be retained for a long time or
discarded quite quickly).
Example systems
* NeonDB branching: https://neon.com/docs/introduction/branching
* Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/
(please see the related blogs on the implementations for the idea).
Ozone can consider supporting this feature. With the rise of the storage
compute separation architecture on databases (OLAP or OLTP), the compute /
caching layer can rely on Ozone as the backing store for agentic workloads
since Ozone supports snapshot and forking. Ozone then can position itself as
the object store for agentic workloads.
This ticket acts as a way to start a discussion in the community on this
direction.
> Support bucket forks for agentic workload
> -----------------------------------------
>
> Key: HDDS-15120
> URL: https://issues.apache.org/jira/browse/HDDS-15120
> Project: Apache Ozone
> Issue Type: New Feature
> Reporter: Ivan Andika
> Priority: Major
>
> Currently, Ozone supports bucket snapshot which creates a read-only immutable
> state of the entire bucket for use cases such as backup, replication,
> compliance, etc.
> However, in the recent rise agentic workloads, there is a need for systems to
> implement forking / branching to cater for multi-agents workload. Unlike
> snapshot, forks can be mutated. The idea of forking and branch is similar to
> Git branch / worktrees where a new "branch" is created based on the base
> directory. Multiple agents can fork the same base file system in parallel and
> mutate these forks without affecting each other. These forks should also have
> zero-copy, similar to snapshot. Additionally, these forks lifetime can varies
> (it can be retained for a long time or discarded quite quickly).
> Example systems
> * NeonDB branching: https://neon.com/docs/introduction/branching
> * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/
> (please see the related blogs on the implementations for the idea).
> Ozone can consider supporting this feature. With the rise of the storage
> compute separation architecture on databases (OLAP or OLTP), the compute /
> caching layer can rely on Ozone as the backing store for agentic workloads
> since Ozone supports snapshot and forking. Ozone then can position itself as
> the object store for agentic workloads.
> This ticket acts as a way to start a discussion in the community on this
> direction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
