[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. This is achieved using the RocksDB checkpoint feature which tracks the current SST files at that point. With the recent rise agentic workloads, there is a need for storage systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations of forks). Ozone can consider supporting this feature. Since more systems implement storage compute separation architecture on object storage, the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, copy-on-write, log subsystems, OM deletions semantic change, etc). was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. This is achieved using the RocksDB checkpoint feature which tracks the current SST files at that point. With the recent rise agentic workloads, there is a need for storage systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations of forks). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, copy-on-write, log subsystems, OM deletions semantic change, etc). > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. This is achieved using the RocksDB checkpoint feature which > tracks the current SST files at that point. > With the recent rise agentic workloads, there is a need for storage systems > to implement forking / branching to cater for multi-agen
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. This is achieved using the RocksDB checkpoint feature which tracks the current SST files at that point. With the recent rise agentic workloads, there is a need for storage systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations of forks). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, copy-on-write, log subsystems, OM deletions semantic change, etc). was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations of forks). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, copy-on-write, log subsystems, OM deletions semantic change, etc). > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. This is achieved using the RocksDB checkpoint feature which > tracks the current SST files at that point. > With the recent rise agentic workloads, there is a need for storage systems > to implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be m
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations of forks). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, copy-on-write, log subsystems, OM deletions semantic change, etc). was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations of forks). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, introducing some kind of log subsystems, OM deletions semantic change, etc). > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. These was achieved using the RocksDB checkpoint feature. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be mutated. The idea of forking and branch is similar to > Git branch / worktrees where a new
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations of forks). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, introducing some kind of log subsystems, OM deletions semantic change, etc). was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, introducing some kind of log subsystems, OM deletions semantic change, etc). > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. These was achieved using the RocksDB checkpoint feature. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be mutated. The idea of forking and branch is similar to > Git branch / worktre
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, introducing some kind of log subsystems, etc). was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. These was achieved using the RocksDB checkpoint feature. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be mutated. The idea of forking and branch is similar to > Git branch / worktrees where a new "branch" is created based on the base > directory. Multiple agents can fork the same base file system in parallel and > mutate these forks without affecting each other. These forks should also have > zero-copy, similar to snapshot (which should only require
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, introducing some kind of log subsystems, OM deletions semantic change, etc). was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot (which should only require O(1) time to create). Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). This might require a radical change of Ozone Manager design (e.g. might need to introduce versioning, reference counting, introducing some kind of log subsystems, etc). > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. These was achieved using the RocksDB checkpoint feature. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be mutated. The idea of forking and branch is similar to > Git branch / worktrees where a new "branch" is
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking or need to write complicated logic to store their forks state). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. These was achieved using the RocksDB checkpoint feature. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be mutated. The idea of forking and branch is similar to > Git branch / worktrees where a new "branch" is created based on the base > directory. Multiple agents can fork the same base file system in parallel and > mutate these forks without affecting each other. These forks should also have > zero-copy, similar to snapshot. Additionally, these forks lifetime can varies > (it can be retained for a long time or discarded quite quickly). > Example systems > * NeonDB branching: https://neon.com/docs/introduction/branching > * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ > (please see the related blogs
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this. was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking). Ozone then can position itself as the object store for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this. > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. These was achieved using the RocksDB checkpoint feature. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be mutated. The idea of forking and branch is similar to > Git branch / worktrees where a new "branch" is created based on the base > directory. Multiple agents can fork the same base file system in parallel and > mutate these forks without affecting each other. These forks should also have > zero-copy, similar to snapshot. Additionally, these forks lifetime can varies > (it can be retained for a long time or discarded quite quickly). > Example systems > * NeonDB branching: https://neon.com/docs/introduction/branching > * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ > (please see the related blogs on the implementations for the idea). > Ozone can consider supporting this feature. With the rise of the storage > compute separation architecture on databases (OLAP or OLTP), the compute / > caching la
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this (and probably try to start prototyping some ideas). was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking). Ozone can then position itself as the open-source object store / distributed file system for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this. > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. These was achieved using the RocksDB checkpoint feature. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be mutated. The idea of forking and branch is similar to > Git branch / worktrees where a new "branch" is created based on the base > directory. Multiple agents can fork the same base file system in parallel and > mutate these forks without affecting each other. These forks should also have > zero-copy, similar to snapshot. Additionally, these forks lifetime can varies > (it can be retained for a long time or discarded quite quickly). > Example systems > * NeonDB branching: https://neon.com/docs/introduction/branching > * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ > (please see the related blogs on the implementations for the idea). > Ozone can consider supporting this feature. With the rise of the storage
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking (they don't need to implement snapshot and forking). Ozone then can position itself as the object store for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this. was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking. Ozone then can position itself as the object store for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this. > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. These was achieved using the RocksDB checkpoint feature. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be mutated. The idea of forking and branch is similar to > Git branch / worktrees where a new "branch" is created based on the base > directory. Multiple agents can fork the same base file system in parallel and > mutate these forks without affecting each other. These forks should also have > zero-copy, similar to snapshot. Additionally, these forks lifetime can varies > (it can be retained for a long time or discarded quite quickly). > Example systems > * NeonDB branching: https://neon.com/docs/introduction/branching > * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ > (please see the related blogs on the implementations for the idea). > Ozone can consider supporting this feature. With the rise of the storage > compute separation architecture on databases (OLAP or OLTP), the compute / > caching layer can rely on Ozone as the backing store for agentic workloads > since Ozone supports sn
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. These was achieved using the RocksDB checkpoint feature. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking. Ozone then can position itself as the object store for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. We can start thinking about this. was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking. Ozone then can position itself as the object store for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. These was achieved using the RocksDB checkpoint feature. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be mutated. The idea of forking and branch is similar to > Git branch / worktrees where a new "branch" is created based on the base > directory. Multiple agents can fork the same base file system in parallel and > mutate these forks without affecting each other. These forks should also have > zero-copy, similar to snapshot. Additionally, these forks lifetime can varies > (it can be retained for a long time or discarded quite quickly). > Example systems > * NeonDB branching: https://neon.com/docs/introduction/branching > * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ > (please see the related blogs on the implementations for the idea). > Ozone can consider supporting this feature. With the rise of the storage > compute separation architecture on databases (OLAP or OLTP), the compute / > caching layer can rely on Ozone as the backing store for agentic workloads > since Ozone supports snapshot and forking. Ozone then can position itself as > the object store for agentic workloads. > This ticket acts as a way to start a discussi
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking. Ozone then can position itself as the object store for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking. Ozone then can position itself as the object store for agentic workloads. This ticket acts as a way to start a discussion on this direction. > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. The idea of > forking and branch is similar to Git branch / worktrees where a new "branch" > is created based on the base directory. Multiple agents can fork the same > base file system in parallel and mutate these forks without affecting each > other. These forks should also have zero-copy, similar to snapshot. > Additionally, these forks lifetime can varies (it can be retained for a long > time or discarded quite quickly). > Example systems > * NeonDB branching: https://neon.com/docs/introduction/branching > * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ > (please see the related blogs on the implementations for the idea). > Ozone can consider supporting this feature. With the rise of the storage > compute separation architecture on databases (OLAP or OLTP), the compute / > caching layer can rely on Ozone as the backing store for agentic workloads > since Ozone supports snapshot and forking. Ozone then can position itself as > the object store for agentic workloads. > This ticket acts as a way to start a discussion in the community on this > direction. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: issues-h...@o
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Description: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. Unlike snapshot, forks can be mutated. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking. Ozone then can position itself as the object store for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. was: Currently, Ozone supports bucket snapshot which creates a read-only immutable state of the entire bucket for use cases such as backup, replication, compliance, etc. However, in the recent rise agentic workloads, there is a need for systems to implement forking / branching to cater for multi-agents workload. The idea of forking and branch is similar to Git branch / worktrees where a new "branch" is created based on the base directory. Multiple agents can fork the same base file system in parallel and mutate these forks without affecting each other. These forks should also have zero-copy, similar to snapshot. Additionally, these forks lifetime can varies (it can be retained for a long time or discarded quite quickly). Example systems * NeonDB branching: https://neon.com/docs/introduction/branching * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ (please see the related blogs on the implementations for the idea). Ozone can consider supporting this feature. With the rise of the storage compute separation architecture on databases (OLAP or OLTP), the compute / caching layer can rely on Ozone as the backing store for agentic workloads since Ozone supports snapshot and forking. Ozone then can position itself as the object store for agentic workloads. This ticket acts as a way to start a discussion in the community on this direction. > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. Unlike > snapshot, forks can be mutated. The idea of forking and branch is similar to > Git branch / worktrees where a new "branch" is created based on the base > directory. Multiple agents can fork the same base file system in parallel and > mutate these forks without affecting each other. These forks should also have > zero-copy, similar to snapshot. Additionally, these forks lifetime can varies > (it can be retained for a long time or discarded quite quickly). > Example systems > * NeonDB branching: https://neon.com/docs/introduction/branching > * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ > (please see the related blogs on the implementations for the idea). > Ozone can consider supporting this feature. With the rise of the storage > compute separation architecture on databases (OLAP or OLTP), the compute / > caching layer can rely on Ozone as the backing store for agentic workloads > since Ozone supports snapshot and forking. Ozone then can position itself as > the object store for agentic workloads. > This ticket acts as a way to start a discussion in the community on this > direction. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubsc
[jira] [Updated] (HDDS-15120) Support bucket forks for agentic workload
[ https://issues.apache.org/jira/browse/HDDS-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated HDDS-15120: --- Summary: Support bucket forks for agentic workload (was: Support bucket forks) > Support bucket forks for agentic workload > - > > Key: HDDS-15120 > URL: https://issues.apache.org/jira/browse/HDDS-15120 > Project: Apache Ozone > Issue Type: New Feature >Reporter: Ivan Andika >Priority: Major > > Currently, Ozone supports bucket snapshot which creates a read-only immutable > state of the entire bucket for use cases such as backup, replication, > compliance, etc. > However, in the recent rise agentic workloads, there is a need for systems to > implement forking / branching to cater for multi-agents workload. The idea of > forking and branch is similar to Git branch / worktrees where a new "branch" > is created based on the base directory. Multiple agents can fork the same > base file system in parallel and mutate these forks without affecting each > other. These forks should also have zero-copy, similar to snapshot. > Additionally, these forks lifetime can varies (it can be retained for a long > time or discarded quite quickly). > Example systems > * NeonDB branching: https://neon.com/docs/introduction/branching > * Tigris Object Store: https://www.tigrisdata.com/docs/snapshots-and-forks/ > (please see the related blogs on the implementations for the idea). > Ozone can consider supporting this feature. With the rise of the storage > compute separation architecture on databases (OLAP or OLTP), the compute / > caching layer can rely on Ozone as the backing store for agentic workloads > since Ozone supports snapshot and forking. Ozone then can position itself as > the object store for agentic workloads. > This ticket acts as a way to start a discussion on this direction. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
