Re: [Gluster-devel] Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes
Hi Jay! I have comments below: - Original Message - From: "Jay Vyas" To: "Joseph Fernandes" Cc: "John Spray" , "Gluster Devel" Sent: Thursday, June 18, 2015 9:33:12 AM Subject: Re: [Gluster-devel] Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes Thanks for annoying letting us know about this so early on; I don't fully understand the plugin functionality So I have some general questions. But am very interested in this. 0) I notice it's largely written in go,compared to the gluster ecosystem which is mostly Python and C. Any reason why? I like go but just curious if there are some technical requirements driving this and if Go might become a more integral part of the gluster core in the future. [LP] Why Go? http://www.g33knotes.org/2014/09/why-i-think-go-provides-excellent.html ;-) .. Maybe GlusterFS will start using Go. But joking aside, it provides an excellent platform for RESTful interfaces and unmatched deployment. 1) is heketi something that would allow things like raid and lvm and so on to be moved from core dependencies into extensions, thus modularizing using glusters core? [LP] Maybe. I'm not sure if heketi should be an orchestrator of storage, but I guess the plugin can do whatever it needs to do. 2) Will heketi actually be co-evolving / working hand in hand with fluster so that some of the storage administration stuff in glusters code base is moved into a broader framework? [LP] I hope so. The goal is for Heketi to provide the framework to allow plugins for storage systems (GlusterFS being one) to accept a set of standard REST commands and expose any others it needs - Luis > On Jun 18, 2015, at 9:07 AM, Joseph Fernandes wrote: > > Agreed we need not be depended ONE technology for the above. > But LVM is a strong contender as a single stable underlying technology that > provides the following. > We can make it plugin based :) . So that ppl who have LVM and are happy with > it can use it. > And we still can have other technology plugins developed in parallel, but let > have Single API standard defined for all. > > ~Joe > > - Original Message - > From: "Jeff Darcy" > To: "Joseph Fernandes" > Cc: "Luis Pabon" , "Gluster Devel" > , "John Spray" > Sent: Thursday, June 18, 2015 5:15:37 PM > Subject: Re: Introducing Heketi: Storage Management Framework with Plugins > for GlusterFS volumes > >> LVM or Volume Manager Dependencies: >> 1) SNAPSHOTS: Gluster snapshots are LVM based > > The current implementation is LVM-centric, which is one reason uptake has > been so low. The intent was always to make it more generic, so that other > mechanisms could be used as well. > > > >> 2) PROVISIONING and ENFORCEMENT: >> As of today Gluster does not have any control on the size of the brick. It >> will consume the brick (xfs)mount point given >> to it without checking on how much it needs to consume. LVM (or any other >> volume manager) will be required to do space provisioning per brick and >> enforce >> limits on size of bricks. > > Some file systems have quota, or we can enforce our own. > >> 3) STORAGE SEGREGATION: >> LVM pools can be used to have storage segregation i.e having primary storage >> pools and secondary(for Gluster replica) pools, >> So that we can crave out proper space from the physical disks attached to >> each node. >> At a high level (i.e Heketi's User) disk space can be viewed as storage >> pools,(i.e by aggreating disk space per pool per node using glusterd) >> To start with we can have Primary pool and secondary pool(for Gluster >> replica) , where each file serving node in the cluster participates in >> these pools via the local LVM pools. > > This functionality in no way depends on LVM. In many cases, mere > subdirectories are sufficient. > >> 4) DATA PROTECTION: >> Further data protection using LVM RAID. Pools can be marked to have RAID >> support on them, courtesy of LVM RAID. >> >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/raid_volumes.html > > Given that we already have replication, erasure coding, etc. many users would > prefer not to reduce storage utilization even further with RAID. Others > would prefer to get the same functionality without LVM, e.g. with ZFS. That's > why RAID has always been - and should remain - optional. > > It's fine that we *can* use LVM features when and where they're available. > Building in *dependencies* on it has been a mistake
Re: [Gluster-devel] Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes
Thanks for annoying letting us know about this so early on; I don't fully understand the plugin functionality So I have some general questions. But am very interested in this. 0) I notice it's largely written in go,compared to the gluster ecosystem which is mostly Python and C. Any reason why? I like go but just curious if there are some technical requirements driving this and if Go might become a more integral part of the gluster core in the future. 1) is heketi something that would allow things like raid and lvm and so on to be moved from core dependencies into extensions, thus modularizing using glusters core? 2) Will heketi actually be co-evolving / working hand in hand with fluster so that some of the storage administration stuff in glusters code base is moved into a broader framework? > On Jun 18, 2015, at 9:07 AM, Joseph Fernandes wrote: > > Agreed we need not be depended ONE technology for the above. > But LVM is a strong contender as a single stable underlying technology that > provides the following. > We can make it plugin based :) . So that ppl who have LVM and are happy with > it can use it. > And we still can have other technology plugins developed in parallel, but let > have Single API standard defined for all. > > ~Joe > > - Original Message - > From: "Jeff Darcy" > To: "Joseph Fernandes" > Cc: "Luis Pabon" , "Gluster Devel" > , "John Spray" > Sent: Thursday, June 18, 2015 5:15:37 PM > Subject: Re: Introducing Heketi: Storage Management Framework with Plugins > for GlusterFS volumes > >> LVM or Volume Manager Dependencies: >> 1) SNAPSHOTS: Gluster snapshots are LVM based > > The current implementation is LVM-centric, which is one reason uptake has > been so low. The intent was always to make it more generic, so that other > mechanisms could be used as well. > > > >> 2) PROVISIONING and ENFORCEMENT: >> As of today Gluster does not have any control on the size of the brick. It >> will consume the brick (xfs)mount point given >> to it without checking on how much it needs to consume. LVM (or any other >> volume manager) will be required to do space provisioning per brick and >> enforce >> limits on size of bricks. > > Some file systems have quota, or we can enforce our own. > >> 3) STORAGE SEGREGATION: >> LVM pools can be used to have storage segregation i.e having primary storage >> pools and secondary(for Gluster replica) pools, >> So that we can crave out proper space from the physical disks attached to >> each node. >> At a high level (i.e Heketi's User) disk space can be viewed as storage >> pools,(i.e by aggreating disk space per pool per node using glusterd) >> To start with we can have Primary pool and secondary pool(for Gluster >> replica) , where each file serving node in the cluster participates in >> these pools via the local LVM pools. > > This functionality in no way depends on LVM. In many cases, mere > subdirectories are sufficient. > >> 4) DATA PROTECTION: >> Further data protection using LVM RAID. Pools can be marked to have RAID >> support on them, courtesy of LVM RAID. >> >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/raid_volumes.html > > Given that we already have replication, erasure coding, etc. many users would > prefer not to reduce storage utilization even further with RAID. Others > would prefer to get the same functionality without LVM, e.g. with ZFS. That's > why RAID has always been - and should remain - optional. > > It's fine that we *can* use LVM features when and where they're available. > Building in *dependencies* on it has been a mistake every time, and repeating > a mistake doesn't make it anything else. > > JOE: > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes
Agreed we need not be depended ONE technology for the above. But LVM is a strong contender as a single stable underlying technology that provides the following. We can make it plugin based :) . So that ppl who have LVM and are happy with it can use it. And we still can have other technology plugins developed in parallel, but let have Single API standard defined for all. ~Joe - Original Message - From: "Jeff Darcy" To: "Joseph Fernandes" Cc: "Luis Pabon" , "Gluster Devel" , "John Spray" Sent: Thursday, June 18, 2015 5:15:37 PM Subject: Re: Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes > LVM or Volume Manager Dependencies: > 1) SNAPSHOTS: Gluster snapshots are LVM based The current implementation is LVM-centric, which is one reason uptake has been so low. The intent was always to make it more generic, so that other mechanisms could be used as well. > 2) PROVISIONING and ENFORCEMENT: > As of today Gluster does not have any control on the size of the brick. It > will consume the brick (xfs)mount point given > to it without checking on how much it needs to consume. LVM (or any other > volume manager) will be required to do space provisioning per brick and > enforce > limits on size of bricks. Some file systems have quota, or we can enforce our own. > 3) STORAGE SEGREGATION: > LVM pools can be used to have storage segregation i.e having primary storage > pools and secondary(for Gluster replica) pools, > So that we can crave out proper space from the physical disks attached to > each node. > At a high level (i.e Heketi's User) disk space can be viewed as storage > pools,(i.e by aggreating disk space per pool per node using glusterd) > To start with we can have Primary pool and secondary pool(for Gluster > replica) , where each file serving node in the cluster participates in > these pools via the local LVM pools. This functionality in no way depends on LVM. In many cases, mere subdirectories are sufficient. > 4) DATA PROTECTION: >Further data protection using LVM RAID. Pools can be marked to have RAID >support on them, courtesy of LVM RAID. > > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/raid_volumes.html Given that we already have replication, erasure coding, etc. many users would prefer not to reduce storage utilization even further with RAID. Others would prefer to get the same functionality without LVM, e.g. with ZFS. That's why RAID has always been - and should remain - optional. It's fine that we *can* use LVM features when and where they're available. Building in *dependencies* on it has been a mistake every time, and repeating a mistake doesn't make it anything else. JOE: ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes
> LVM or Volume Manager Dependencies: > 1) SNAPSHOTS: Gluster snapshots are LVM based The current implementation is LVM-centric, which is one reason uptake has been so low. The intent was always to make it more generic, so that other mechanisms could be used as well. > 2) PROVISIONING and ENFORCEMENT: > As of today Gluster does not have any control on the size of the brick. It > will consume the brick (xfs)mount point given > to it without checking on how much it needs to consume. LVM (or any other > volume manager) will be required to do space provisioning per brick and > enforce > limits on size of bricks. Some file systems have quota, or we can enforce our own. > 3) STORAGE SEGREGATION: > LVM pools can be used to have storage segregation i.e having primary storage > pools and secondary(for Gluster replica) pools, > So that we can crave out proper space from the physical disks attached to > each node. > At a high level (i.e Heketi's User) disk space can be viewed as storage > pools,(i.e by aggreating disk space per pool per node using glusterd) > To start with we can have Primary pool and secondary pool(for Gluster > replica) , where each file serving node in the cluster participates in > these pools via the local LVM pools. This functionality in no way depends on LVM. In many cases, mere subdirectories are sufficient. > 4) DATA PROTECTION: >Further data protection using LVM RAID. Pools can be marked to have RAID >support on them, courtesy of LVM RAID. > > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/raid_volumes.html Given that we already have replication, erasure coding, etc. many users would prefer not to reduce storage utilization even further with RAID. Others would prefer to get the same functionality without LVM, e.g. with ZFS. That's why RAID has always been - and should remain - optional. It's fine that we *can* use LVM features when and where they're available. Building in *dependencies* on it has been a mistake every time, and repeating a mistake doesn't make it anything else. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes
LVM or Volume Manager Dependencies: 1) SNAPSHOTS: Gluster snapshots are LVM based 2) PROVISIONING and ENFORCEMENT: As of today Gluster does not have any control on the size of the brick. It will consume the brick (xfs)mount point given to it without checking on how much it needs to consume. LVM (or any other volume manager) will be required to do space provisioning per brick and enforce limits on size of bricks. 3) STORAGE SEGREGATION: LVM pools can be used to have storage segregation i.e having primary storage pools and secondary(for Gluster replica) pools, So that we can crave out proper space from the physical disks attached to each node. At a high level (i.e Heketi's User) disk space can be viewed as storage pools,(i.e by aggreating disk space per pool per node using glusterd) To start with we can have Primary pool and secondary pool(for Gluster replica) , where each file serving node in the cluster participates in these pools via the local LVM pools. 4) DATA PROTECTION: Further data protection using LVM RAID. Pools can be marked to have RAID support on them, courtesy of LVM RAID. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/raid_volumes.html ~Joe - Original Message - From: "Luis Pabon" To: "Jeff Darcy" Cc: "John Spray" , storage-...@redhat.com Sent: Thursday, June 18, 2015 8:05:43 AM Subject: Re: Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes Currently, Heketi is setup to provide volumes to Manila which requires them to have snapshot capabilities. AFAIK, the only way to do that is to use LVM backed bricks in GlusterFS. Heketi itself does not depend on LVM to create bricks and can be enhanced to support LVM-less nodes. - Luis - Original Message - From: "Jeff Darcy" To: "Luis Pabon" Cc: "John Spray" , storage-...@redhat.com Sent: Wednesday, June 17, 2015 7:13:38 PM Subject: Re: Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes > [LP] In Manila they are called 'shares', but for simplicity, I called them > 'volumes' as a generic mountable network file system (NFS, CIFS, GlusterFS, > whatever). The plugin for glusterfs does create fresh "bricks" for each > volume, but that is how glusterfs works. The plugin creates these bricks on > top of thinly provisioned LVs, which the plugin manages. Does this imply a dependency on LVM? --- Note: This list is intended for discussions relating to Red Hat Storage products, customers and/or support. Discussions on GlusterFS and Ceph architecture, design and engineering should go to relevant upstream mailing lists. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes
Great! I look forward to it. In the near term, we are planning on using
OpenStack Swift's Ring to determine brick placement in the cluster. I look
forward to your plan and see how we can work together.. (Maybe Glusterd-2.0 is
going to be written in Go and we can share code, or even integrate heketi into
glusterd ;-) ).
- Luis
- Original Message -
From: "Kaushal M"
To: "Luis Pabon"
Cc: "gluster-devel@gluster.org >> Gluster Devel"
Sent: Tuesday, June 16, 2015 5:19:02 AM
Subject: Re: [Gluster-devel] Introducing Heketi: Storage Management Framework
with Plugins for GlusterFS volumes
Hey Luis,
You just announced this right when we were about to make public our
plans for Manila integration w.r.t GlusterD-2.0.
We are about to kickstart GlusterD-2.0, and the first piece we wanted
to implement was a method to create volumes intelligently. This is
pretty much what heketi is attempting to do now, so maybe we could
collaborate.
I'll be publishing our full plan later today. Let me know what you
think about it.
Cheers.
~kaushal
On Mon, Jun 15, 2015 at 9:31 PM, Luis Pabon wrote:
> Hi all,
>
> As we continue forward with OpenStack Manila and Kubernetes integration
> with GlusterFS, we require a simpler method of managing volumes and bricks.
> We introduce Heketi ( https://github.com/heketi ), a RESTful storage
> management framework which enables certain storage systems to be easily
> managed. Heketi poses itself not an abstraction layer, but instead as a
> framework for developers to enhance their existing storage system management
> through a plugin interface. Heketi also provides a RESTful interface which
> enables callers to easily manage the storage system. For GlusterFS,
> currently we only plan on managing the creation and deletion of volumes in
> GlusterFS through a simple RESTful interface. Heketi will also manage the
> brick location and life cycle.
>
> Heketi allows users to create volumes with a simple command and not worry
> about which bricks to use. A simple command to create a 1 TB volume would
> look like this:
>
> POST on http:///volumes, with JSON { "size" : 10 }
>
> To delete the volume:
>
> DELETE on http:///volumes/
>
>
> * A demo is available https://github.com/heketi/vagrant-heketi , so please
> try it out and let us know what you think.
>
>
> ** Heketi is still quite young, but the plan is to finish by early September.
>
> Please let us know what you think.
>
> - Luis
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes
Hey Luis,
You just announced this right when we were about to make public our
plans for Manila integration w.r.t GlusterD-2.0.
We are about to kickstart GlusterD-2.0, and the first piece we wanted
to implement was a method to create volumes intelligently. This is
pretty much what heketi is attempting to do now, so maybe we could
collaborate.
I'll be publishing our full plan later today. Let me know what you
think about it.
Cheers.
~kaushal
On Mon, Jun 15, 2015 at 9:31 PM, Luis Pabon wrote:
> Hi all,
>
> As we continue forward with OpenStack Manila and Kubernetes integration
> with GlusterFS, we require a simpler method of managing volumes and bricks.
> We introduce Heketi ( https://github.com/heketi ), a RESTful storage
> management framework which enables certain storage systems to be easily
> managed. Heketi poses itself not an abstraction layer, but instead as a
> framework for developers to enhance their existing storage system management
> through a plugin interface. Heketi also provides a RESTful interface which
> enables callers to easily manage the storage system. For GlusterFS,
> currently we only plan on managing the creation and deletion of volumes in
> GlusterFS through a simple RESTful interface. Heketi will also manage the
> brick location and life cycle.
>
> Heketi allows users to create volumes with a simple command and not worry
> about which bricks to use. A simple command to create a 1 TB volume would
> look like this:
>
> POST on http:///volumes, with JSON { "size" : 10 }
>
> To delete the volume:
>
> DELETE on http:///volumes/
>
>
> * A demo is available https://github.com/heketi/vagrant-heketi , so please
> try it out and let us know what you think.
>
>
> ** Heketi is still quite young, but the plan is to finish by early September.
>
> Please let us know what you think.
>
> - Luis
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes
Awesome, this seems to the filling a nice gap, a feature for which was
opened long back (Sept. 2014) [1]
Will checkout the demo soon.
thanx,
deepak
[1]:
http://www.gluster.org/community/documentation/index.php/Features/glusterd-intelligent-volume-creation
On Mon, Jun 15, 2015 at 9:31 PM, Luis Pabon wrote:
> Hi all,
>
> As we continue forward with OpenStack Manila and Kubernetes integration
> with GlusterFS, we require a simpler method of managing volumes and
> bricks. We introduce Heketi ( https://github.com/heketi ), a RESTful
> storage management framework which enables certain storage systems to be
> easily managed. Heketi poses itself not an abstraction layer, but instead
> as a framework for developers to enhance their existing storage system
> management through a plugin interface. Heketi also provides a RESTful
> interface which enables callers to easily manage the storage system. For
> GlusterFS, currently we only plan on managing the creation and deletion of
> volumes in GlusterFS through a simple RESTful interface. Heketi will also
> manage the brick location and life cycle.
>
> Heketi allows users to create volumes with a simple command and not worry
> about which bricks to use. A simple command to create a 1 TB volume would
> look like this:
>
> POST on http:///volumes, with JSON { "size" : 10 }
>
> To delete the volume:
>
> DELETE on http:///volumes/
>
>
> * A demo is available https://github.com/heketi/vagrant-heketi , so
> please try it out and let us know what you think.
>
>
> ** Heketi is still quite young, but the plan is to finish by early
> September.
>
> Please let us know what you think.
>
> - Luis
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Introducing Heketi: Storage Management Framework with Plugins for GlusterFS volumes
Hi all,
As we continue forward with OpenStack Manila and Kubernetes integration with
GlusterFS, we require a simpler method of managing volumes and bricks. We
introduce Heketi ( https://github.com/heketi ), a RESTful storage management
framework which enables certain storage systems to be easily managed. Heketi
poses itself not an abstraction layer, but instead as a framework for
developers to enhance their existing storage system management through a plugin
interface. Heketi also provides a RESTful interface which enables callers to
easily manage the storage system. For GlusterFS, currently we only plan on
managing the creation and deletion of volumes in GlusterFS through a simple
RESTful interface. Heketi will also manage the brick location and life cycle.
Heketi allows users to create volumes with a simple command and not worry about
which bricks to use. A simple command to create a 1 TB volume would look like
this:
POST on http:///volumes, with JSON { "size" : 10 }
To delete the volume:
DELETE on http:///volumes/
* A demo is available https://github.com/heketi/vagrant-heketi , so please try
it out and let us know what you think.
** Heketi is still quite young, but the plan is to finish by early September.
Please let us know what you think.
- Luis
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
