Re: [Gluster-devel] [Heketi] Block store related API design discussion
Awesome, Thanks guys. - Luis - Original Message - From: "Pranith Kumar Karampuri" To: "Niels de Vos" Cc: "Luis Pabón" , "gluster-devel" , "Stephen Watt" , "Ramakrishna Yekulla" , "Humble Chirammal" Sent: Tuesday, September 20, 2016 5:53:30 AM Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion On Mon, Sep 19, 2016 at 9:22 PM, Niels de Vos wrote: > On Mon, Sep 19, 2016 at 10:31:11AM -0400, Luis Pabón wrote: > > Using qemu is interesting, but the I/O should be using the IO path of > QEMU block API. If not, > > TCMU would not know how to work with QEMU dynamic QCOW2 files. > > > > Now, if TCMU already has this, then that would be great! > > It has a qcow2 header, maybe you guys are lucky! > https://github.com/open-iscsi/tcmu-runner/blob/master/qcow2.h Sent the earlier mail before seeing this mail :-). So yes, what we discussed is to see if this qemu in tcmu can internally use gfapi for doing the operations or not is something we are trying to find out. > > > Niels > > > > > - Luis > > > > - Original Message - > > From: "Prasanna Kalever" > > To: "Niels de Vos" > > Cc: "Luis Pabón" , "Stephen Watt" , > "gluster-devel" , "Ramakrishna Yekulla" < > rre...@redhat.com>, "Humble Chirammal" > > Sent: Monday, September 19, 2016 7:13:36 AM > > Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > > > > On Mon, Sep 19, 2016 at 4:09 PM, Niels de Vos wrote: > > > > > > On Mon, Sep 19, 2016 at 03:34:29PM +0530, Prasanna Kalever wrote: > > > > On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos > wrote: > > > > > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: > > > > >> Very good points. Thanks Prasanna for putting this together. I > agree with > > > > >> your comments in that Heketi is the high level abstraction API > and it should have > > > > >> an API similar of what is described by Prasanna. > > > > >> > > > > >> I definitely do not think any File Api should be available in > Heketi, > > > > >> because that is an implementation of the Block API. The Heketi > API should > > > > >> be similar to something like OpenStack Cinder. > > > > >> > > > > >> I think that the actual management of the Volumes used for Block > storage > > > > >> and the files in them should be all managed by Heketi. How they > are > > > > >> actually created is still to be determined, but we could have > Heketi > > > > >> create them, or have helper programs do that. > > > > > > > > > > Maybe a tool like qemu-img? If whatever iscsi service understand > the > > > > > format (at the very least 'raw'), you could get functionality like > > > > > snapshots pretty simple. > > > > > > > > Niels, > > > > > > > > This is brilliant and subset of the Idea falls in one among my > > > > thoughts, only concern is about building dependencies of qemu with > > > > Heketi. > > > > But at an advantage of easy and cool snapshots solution. > > > > > > And well tested as I understand that oVirt is moving to use qemu-img as > > > well. Other tools are able to use the qcow2 format, maybe the iscsi > > > servce that gets used does so too. > > > > > > Has there already been a decision on what Heketi will configure as > iscsi > > > service? I am aware of the tgt [1] and LIO/TCMU [2] projects. > > > > Niels, > > > > yes we will be using TCMU (Kernel Module) and TCMU-runner (user space > > service) to expose file in Gluster volume as an iSCSI target. > > more at [1], [2] & [3] > > > > [1] https://pkalever.wordpress.com/2016/06/23/gluster- > solution-for-non-shared-persistent-storage-in-docker-container/ > > [2] https://pkalever.wordpress.com/2016/06/29/non-shared- > persistent-gluster-storage-with-kubernetes/ > > [3] https://pkalever.wordpress.com/2016/08/16/read-write- > once-persistent-storage-for-openshift-origin-using-gluster/ > > > > -- > > Prasanna > > > > > > > > Niels > > > > > > 1. http://stgt.sourceforge.net/ > > > 2. https://github.com/open-iscsi/tcmu-runner > > >http://blog.gluster.org/2016/04/using-lio-with-gluster/ > > > > >
Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Mon, Sep 19, 2016 at 9:22 PM, Niels de Vos wrote: > On Mon, Sep 19, 2016 at 10:31:11AM -0400, Luis Pabón wrote: > > Using qemu is interesting, but the I/O should be using the IO path of > QEMU block API. If not, > > TCMU would not know how to work with QEMU dynamic QCOW2 files. > > > > Now, if TCMU already has this, then that would be great! > > It has a qcow2 header, maybe you guys are lucky! > https://github.com/open-iscsi/tcmu-runner/blob/master/qcow2.h Sent the earlier mail before seeing this mail :-). So yes, what we discussed is to see if this qemu in tcmu can internally use gfapi for doing the operations or not is something we are trying to find out. > > > Niels > > > > > - Luis > > > > - Original Message - > > From: "Prasanna Kalever" > > To: "Niels de Vos" > > Cc: "Luis Pabón" , "Stephen Watt" , > "gluster-devel" , "Ramakrishna Yekulla" < > rre...@redhat.com>, "Humble Chirammal" > > Sent: Monday, September 19, 2016 7:13:36 AM > > Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > > > > On Mon, Sep 19, 2016 at 4:09 PM, Niels de Vos wrote: > > > > > > On Mon, Sep 19, 2016 at 03:34:29PM +0530, Prasanna Kalever wrote: > > > > On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos > wrote: > > > > > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: > > > > >> Very good points. Thanks Prasanna for putting this together. I > agree with > > > > >> your comments in that Heketi is the high level abstraction API > and it should have > > > > >> an API similar of what is described by Prasanna. > > > > >> > > > > >> I definitely do not think any File Api should be available in > Heketi, > > > > >> because that is an implementation of the Block API. The Heketi > API should > > > > >> be similar to something like OpenStack Cinder. > > > > >> > > > > >> I think that the actual management of the Volumes used for Block > storage > > > > >> and the files in them should be all managed by Heketi. How they > are > > > > >> actually created is still to be determined, but we could have > Heketi > > > > >> create them, or have helper programs do that. > > > > > > > > > > Maybe a tool like qemu-img? If whatever iscsi service understand > the > > > > > format (at the very least 'raw'), you could get functionality like > > > > > snapshots pretty simple. > > > > > > > > Niels, > > > > > > > > This is brilliant and subset of the Idea falls in one among my > > > > thoughts, only concern is about building dependencies of qemu with > > > > Heketi. > > > > But at an advantage of easy and cool snapshots solution. > > > > > > And well tested as I understand that oVirt is moving to use qemu-img as > > > well. Other tools are able to use the qcow2 format, maybe the iscsi > > > servce that gets used does so too. > > > > > > Has there already been a decision on what Heketi will configure as > iscsi > > > service? I am aware of the tgt [1] and LIO/TCMU [2] projects. > > > > Niels, > > > > yes we will be using TCMU (Kernel Module) and TCMU-runner (user space > > service) to expose file in Gluster volume as an iSCSI target. > > more at [1], [2] & [3] > > > > [1] https://pkalever.wordpress.com/2016/06/23/gluster- > solution-for-non-shared-persistent-storage-in-docker-container/ > > [2] https://pkalever.wordpress.com/2016/06/29/non-shared- > persistent-gluster-storage-with-kubernetes/ > > [3] https://pkalever.wordpress.com/2016/08/16/read-write- > once-persistent-storage-for-openshift-origin-using-gluster/ > > > > -- > > Prasanna > > > > > > > > Niels > > > > > > 1. http://stgt.sourceforge.net/ > > > 2. https://github.com/open-iscsi/tcmu-runner > > >http://blog.gluster.org/2016/04/using-lio-with-gluster/ > > > > > > > > > > > -- > > > > Prasanna > > > > > > > > > > > > > > Niels > > > > > > > > > > > > > > >> We also need to document the exact workflow to enable a file in > > > > >> a Gluster volume to be exposed as a block device. This will help > > > > >>
Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos wrote: > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: > > Very good points. Thanks Prasanna for putting this together. I agree > with > > your comments in that Heketi is the high level abstraction API and it > should have > > an API similar of what is described by Prasanna. > > > > I definitely do not think any File Api should be available in Heketi, > > because that is an implementation of the Block API. The Heketi API > should > > be similar to something like OpenStack Cinder. > > > > I think that the actual management of the Volumes used for Block storage > > and the files in them should be all managed by Heketi. How they are > > actually created is still to be determined, but we could have Heketi > > create them, or have helper programs do that. > > Maybe a tool like qemu-img? If whatever iscsi service understand the > format (at the very least 'raw'), you could get functionality like > snapshots pretty simple. > Prasanna, Poornima and I just discussed about this. Prasanna is doing this experiment to see if we can use qcow from tcmu-runner to get this piece working. If yes, we definitely will get snapshots for free :-). Prasanna will confirm it based on his experiments. > > Niels > > > > We also need to document the exact workflow to enable a file in > > a Gluster volume to be exposed as a block device. This will help > > determine where the creation of the file could take place. > > > > We can capture our decisions from these discussions in the > > following page: > > > > https://github.com/heketi/heketi/wiki/Proposed-Changes > > > > - Luis > > > > > > - Original Message - > > From: "Humble Chirammal" > > To: "Raghavendra Talur" > > Cc: "Prasanna Kalever" , "gluster-devel" < > gluster-devel@gluster.org>, "Stephen Watt" , "Luis > Pabon" , "Michael Adam" , > "Ramakrishna Yekulla" , "Mohamed Ashiq Liyazudeen" < > mliya...@redhat.com> > > Sent: Tuesday, September 13, 2016 2:23:39 AM > > Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > > > > > > > > > > > > ----- Original Message ----- > > | From: "Raghavendra Talur" > > | To: "Prasanna Kalever" > > | Cc: "gluster-devel" , "Stephen Watt" < > sw...@redhat.com>, "Luis Pabon" , > > | "Michael Adam" , "Humble Chirammal" < > hchir...@redhat.com>, "Ramakrishna Yekulla" > > | , "Mohamed Ashiq Liyazudeen" > > | Sent: Tuesday, September 13, 2016 11:08:44 AM > > | Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > > | > > | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever < > pkale...@redhat.com> > > | wrote: > > | > > | > Hi all, > > | > > > | > This mail is open for discussion on gluster block store integration > with > > | > heketi and its REST API interface design constraints. > > | > > > | > > > | > ___ Volume Request ... > > | > | > > | > | > > | > PVC claim -> Heketi --->| > > | > | > > | > | > > | > | > > | > | > > | > |__ BlockCreate > > | > | | > > | > | |__ BlockInfo > > | > | | > > | > |___ Block Request (APIS)-> |__ BlockResize > > | > | > > | > |__ BlockList > > | > | > > | > |__ BlockDelete > > | > > > | > Heketi will have block API and volume API, when user submit a > Persistent > > | > volume claim, Kubernetes provisioner based on the storage class(from > PVC) > > | > talks to heketi for storage, heketi intern calls block or volume > API's > > | > based on request. > > | > > > | > > | This is probably wrong. It won't be Heketi calling block or
Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Mon, Sep 19, 2016 at 10:31:11AM -0400, Luis Pabón wrote: > Using qemu is interesting, but the I/O should be using the IO path of QEMU > block API. If not, > TCMU would not know how to work with QEMU dynamic QCOW2 files. > > Now, if TCMU already has this, then that would be great! It has a qcow2 header, maybe you guys are lucky! https://github.com/open-iscsi/tcmu-runner/blob/master/qcow2.h Niels > > - Luis > > - Original Message - > From: "Prasanna Kalever" > To: "Niels de Vos" > Cc: "Luis Pabón" , "Stephen Watt" , > "gluster-devel" , "Ramakrishna Yekulla" > , "Humble Chirammal" > Sent: Monday, September 19, 2016 7:13:36 AM > Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > > On Mon, Sep 19, 2016 at 4:09 PM, Niels de Vos wrote: > > > > On Mon, Sep 19, 2016 at 03:34:29PM +0530, Prasanna Kalever wrote: > > > On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos wrote: > > > > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: > > > >> Very good points. Thanks Prasanna for putting this together. I agree > > > >> with > > > >> your comments in that Heketi is the high level abstraction API and it > > > >> should have > > > >> an API similar of what is described by Prasanna. > > > >> > > > >> I definitely do not think any File Api should be available in Heketi, > > > >> because that is an implementation of the Block API. The Heketi API > > > >> should > > > >> be similar to something like OpenStack Cinder. > > > >> > > > >> I think that the actual management of the Volumes used for Block > > > >> storage > > > >> and the files in them should be all managed by Heketi. How they are > > > >> actually created is still to be determined, but we could have Heketi > > > >> create them, or have helper programs do that. > > > > > > > > Maybe a tool like qemu-img? If whatever iscsi service understand the > > > > format (at the very least 'raw'), you could get functionality like > > > > snapshots pretty simple. > > > > > > Niels, > > > > > > This is brilliant and subset of the Idea falls in one among my > > > thoughts, only concern is about building dependencies of qemu with > > > Heketi. > > > But at an advantage of easy and cool snapshots solution. > > > > And well tested as I understand that oVirt is moving to use qemu-img as > > well. Other tools are able to use the qcow2 format, maybe the iscsi > > servce that gets used does so too. > > > > Has there already been a decision on what Heketi will configure as iscsi > > service? I am aware of the tgt [1] and LIO/TCMU [2] projects. > > Niels, > > yes we will be using TCMU (Kernel Module) and TCMU-runner (user space > service) to expose file in Gluster volume as an iSCSI target. > more at [1], [2] & [3] > > [1] > https://pkalever.wordpress.com/2016/06/23/gluster-solution-for-non-shared-persistent-storage-in-docker-container/ > [2] > https://pkalever.wordpress.com/2016/06/29/non-shared-persistent-gluster-storage-with-kubernetes/ > [3] > https://pkalever.wordpress.com/2016/08/16/read-write-once-persistent-storage-for-openshift-origin-using-gluster/ > > -- > Prasanna > > > > > Niels > > > > 1. http://stgt.sourceforge.net/ > > 2. https://github.com/open-iscsi/tcmu-runner > >http://blog.gluster.org/2016/04/using-lio-with-gluster/ > > > > > > > > -- > > > Prasanna > > > > > > > > > > > Niels > > > > > > > > > > > >> We also need to document the exact workflow to enable a file in > > > >> a Gluster volume to be exposed as a block device. This will help > > > >> determine where the creation of the file could take place. > > > >> > > > >> We can capture our decisions from these discussions in the > > > >> following page: > > > >> > > > >> https://github.com/heketi/heketi/wiki/Proposed-Changes > > > >> > > > >> - Luis > > > >> > > > >> > > > >> - Original Message - > > > >> From: "Humble Chirammal" > > > >> To: "Raghavendra Talur" > > > >> Cc: "Prasanna Kalever" , "gluster-devel"
Re: [Gluster-devel] [Heketi] Block store related API design discussion
Using qemu is interesting, but the I/O should be using the IO path of QEMU block API. If not, TCMU would not know how to work with QEMU dynamic QCOW2 files. Now, if TCMU already has this, then that would be great! - Luis - Original Message - From: "Prasanna Kalever" To: "Niels de Vos" Cc: "Luis Pabón" , "Stephen Watt" , "gluster-devel" , "Ramakrishna Yekulla" , "Humble Chirammal" Sent: Monday, September 19, 2016 7:13:36 AM Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion On Mon, Sep 19, 2016 at 4:09 PM, Niels de Vos wrote: > > On Mon, Sep 19, 2016 at 03:34:29PM +0530, Prasanna Kalever wrote: > > On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos wrote: > > > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: > > >> Very good points. Thanks Prasanna for putting this together. I agree > > >> with > > >> your comments in that Heketi is the high level abstraction API and it > > >> should have > > >> an API similar of what is described by Prasanna. > > >> > > >> I definitely do not think any File Api should be available in Heketi, > > >> because that is an implementation of the Block API. The Heketi API > > >> should > > >> be similar to something like OpenStack Cinder. > > >> > > >> I think that the actual management of the Volumes used for Block storage > > >> and the files in them should be all managed by Heketi. How they are > > >> actually created is still to be determined, but we could have Heketi > > >> create them, or have helper programs do that. > > > > > > Maybe a tool like qemu-img? If whatever iscsi service understand the > > > format (at the very least 'raw'), you could get functionality like > > > snapshots pretty simple. > > > > Niels, > > > > This is brilliant and subset of the Idea falls in one among my > > thoughts, only concern is about building dependencies of qemu with > > Heketi. > > But at an advantage of easy and cool snapshots solution. > > And well tested as I understand that oVirt is moving to use qemu-img as > well. Other tools are able to use the qcow2 format, maybe the iscsi > servce that gets used does so too. > > Has there already been a decision on what Heketi will configure as iscsi > service? I am aware of the tgt [1] and LIO/TCMU [2] projects. Niels, yes we will be using TCMU (Kernel Module) and TCMU-runner (user space service) to expose file in Gluster volume as an iSCSI target. more at [1], [2] & [3] [1] https://pkalever.wordpress.com/2016/06/23/gluster-solution-for-non-shared-persistent-storage-in-docker-container/ [2] https://pkalever.wordpress.com/2016/06/29/non-shared-persistent-gluster-storage-with-kubernetes/ [3] https://pkalever.wordpress.com/2016/08/16/read-write-once-persistent-storage-for-openshift-origin-using-gluster/ -- Prasanna > > Niels > > 1. http://stgt.sourceforge.net/ > 2. https://github.com/open-iscsi/tcmu-runner >http://blog.gluster.org/2016/04/using-lio-with-gluster/ > > > > > -- > > Prasanna > > > > > > > > Niels > > > > > > > > >> We also need to document the exact workflow to enable a file in > > >> a Gluster volume to be exposed as a block device. This will help > > >> determine where the creation of the file could take place. > > >> > > >> We can capture our decisions from these discussions in the > > >> following page: > > >> > > >> https://github.com/heketi/heketi/wiki/Proposed-Changes > > >> > > >> - Luis > > >> > > >> > > >> - Original Message - > > >> From: "Humble Chirammal" > > >> To: "Raghavendra Talur" > > >> Cc: "Prasanna Kalever" , "gluster-devel" > > >> , "Stephen Watt" , "Luis > > >> Pabon" , "Michael Adam" , > > >> "Ramakrishna Yekulla" , "Mohamed Ashiq Liyazudeen" > > >> > > >> Sent: Tuesday, September 13, 2016 2:23:39 AM > > >> Subject: Re: [Gluster-devel] [Heketi] Block store related API design > > >> discussion > > >> > > >> > > >> > > >> > > >> > > >> - Ori
Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Mon, Sep 19, 2016 at 4:09 PM, Niels de Vos wrote: > > On Mon, Sep 19, 2016 at 03:34:29PM +0530, Prasanna Kalever wrote: > > On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos wrote: > > > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: > > >> Very good points. Thanks Prasanna for putting this together. I agree > > >> with > > >> your comments in that Heketi is the high level abstraction API and it > > >> should have > > >> an API similar of what is described by Prasanna. > > >> > > >> I definitely do not think any File Api should be available in Heketi, > > >> because that is an implementation of the Block API. The Heketi API > > >> should > > >> be similar to something like OpenStack Cinder. > > >> > > >> I think that the actual management of the Volumes used for Block storage > > >> and the files in them should be all managed by Heketi. How they are > > >> actually created is still to be determined, but we could have Heketi > > >> create them, or have helper programs do that. > > > > > > Maybe a tool like qemu-img? If whatever iscsi service understand the > > > format (at the very least 'raw'), you could get functionality like > > > snapshots pretty simple. > > > > Niels, > > > > This is brilliant and subset of the Idea falls in one among my > > thoughts, only concern is about building dependencies of qemu with > > Heketi. > > But at an advantage of easy and cool snapshots solution. > > And well tested as I understand that oVirt is moving to use qemu-img as > well. Other tools are able to use the qcow2 format, maybe the iscsi > servce that gets used does so too. > > Has there already been a decision on what Heketi will configure as iscsi > service? I am aware of the tgt [1] and LIO/TCMU [2] projects. Niels, yes we will be using TCMU (Kernel Module) and TCMU-runner (user space service) to expose file in Gluster volume as an iSCSI target. more at [1], [2] & [3] [1] https://pkalever.wordpress.com/2016/06/23/gluster-solution-for-non-shared-persistent-storage-in-docker-container/ [2] https://pkalever.wordpress.com/2016/06/29/non-shared-persistent-gluster-storage-with-kubernetes/ [3] https://pkalever.wordpress.com/2016/08/16/read-write-once-persistent-storage-for-openshift-origin-using-gluster/ -- Prasanna > > Niels > > 1. http://stgt.sourceforge.net/ > 2. https://github.com/open-iscsi/tcmu-runner >http://blog.gluster.org/2016/04/using-lio-with-gluster/ > > > > > -- > > Prasanna > > > > > > > > Niels > > > > > > > > >> We also need to document the exact workflow to enable a file in > > >> a Gluster volume to be exposed as a block device. This will help > > >> determine where the creation of the file could take place. > > >> > > >> We can capture our decisions from these discussions in the > > >> following page: > > >> > > >> https://github.com/heketi/heketi/wiki/Proposed-Changes > > >> > > >> - Luis > > >> > > >> > > >> - Original Message - > > >> From: "Humble Chirammal" > > >> To: "Raghavendra Talur" > > >> Cc: "Prasanna Kalever" , "gluster-devel" > > >> , "Stephen Watt" , "Luis > > >> Pabon" , "Michael Adam" , > > >> "Ramakrishna Yekulla" , "Mohamed Ashiq Liyazudeen" > > >> > > >> Sent: Tuesday, September 13, 2016 2:23:39 AM > > >> Subject: Re: [Gluster-devel] [Heketi] Block store related API design > > >> discussion > > >> > > >> > > >> > > >> > > >> > > >> - Original Message - > > >> | From: "Raghavendra Talur" > > >> | To: "Prasanna Kalever" > > >> | Cc: "gluster-devel" , "Stephen Watt" > > >> , "Luis Pabon" , > > >> | "Michael Adam" , "Humble Chirammal" > > >> , "Ramakrishna Yekulla" > > >> | , "Mohamed Ashiq Liyazudeen" > > >> | Sent: Tuesday, September 13, 2016 11:08:44 AM > > >> | Subject: Re: [Gluster-devel] [Heketi] Block store related API design > > >> discussion > > >> | > > >> | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever
Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Mon, Sep 19, 2016 at 03:34:29PM +0530, Prasanna Kalever wrote: > On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos wrote: > > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: > >> Very good points. Thanks Prasanna for putting this together. I agree with > >> your comments in that Heketi is the high level abstraction API and it > >> should have > >> an API similar of what is described by Prasanna. > >> > >> I definitely do not think any File Api should be available in Heketi, > >> because that is an implementation of the Block API. The Heketi API should > >> be similar to something like OpenStack Cinder. > >> > >> I think that the actual management of the Volumes used for Block storage > >> and the files in them should be all managed by Heketi. How they are > >> actually created is still to be determined, but we could have Heketi > >> create them, or have helper programs do that. > > > > Maybe a tool like qemu-img? If whatever iscsi service understand the > > format (at the very least 'raw'), you could get functionality like > > snapshots pretty simple. > > Niels, > > This is brilliant and subset of the Idea falls in one among my > thoughts, only concern is about building dependencies of qemu with > Heketi. > But at an advantage of easy and cool snapshots solution. And well tested as I understand that oVirt is moving to use qemu-img as well. Other tools are able to use the qcow2 format, maybe the iscsi servce that gets used does so too. Has there already been a decision on what Heketi will configure as iscsi service? I am aware of the tgt [1] and LIO/TCMU [2] projects. Niels 1. http://stgt.sourceforge.net/ 2. https://github.com/open-iscsi/tcmu-runner http://blog.gluster.org/2016/04/using-lio-with-gluster/ > > -- > Prasanna > > > > > Niels > > > > > >> We also need to document the exact workflow to enable a file in > >> a Gluster volume to be exposed as a block device. This will help > >> determine where the creation of the file could take place. > >> > >> We can capture our decisions from these discussions in the > >> following page: > >> > >> https://github.com/heketi/heketi/wiki/Proposed-Changes > >> > >> - Luis > >> > >> > >> - Original Message - > >> From: "Humble Chirammal" > >> To: "Raghavendra Talur" > >> Cc: "Prasanna Kalever" , "gluster-devel" > >> , "Stephen Watt" , "Luis > >> Pabon" , "Michael Adam" , > >> "Ramakrishna Yekulla" , "Mohamed Ashiq Liyazudeen" > >> > >> Sent: Tuesday, September 13, 2016 2:23:39 AM > >> Subject: Re: [Gluster-devel] [Heketi] Block store related API design > >> discussion > >> > >> > >> > >> > >> > >> - Original Message - > >> | From: "Raghavendra Talur" > >> | To: "Prasanna Kalever" > >> | Cc: "gluster-devel" , "Stephen Watt" > >> , "Luis Pabon" , > >> | "Michael Adam" , "Humble Chirammal" > >> , "Ramakrishna Yekulla" > >> | , "Mohamed Ashiq Liyazudeen" > >> | Sent: Tuesday, September 13, 2016 11:08:44 AM > >> | Subject: Re: [Gluster-devel] [Heketi] Block store related API design > >> discussion > >> | > >> | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever > >> | wrote: > >> | > >> | > Hi all, > >> | > > >> | > This mail is open for discussion on gluster block store integration > >> with > >> | > heketi and its REST API interface design constraints. > >> | > > >> | > > >> | > ___ Volume Request ... > >> | > | > >> | > | > >> | > PVC claim -> Heketi --->| > >> | > | > >> | > | > >> | > | > >> | > | > >> | > |__ BlockCreate > >> | > | | > >> | > | |__ BlockInfo > >> | > | | > >> | > |___ Block Request (APIS)->
Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos wrote: > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: >> Very good points. Thanks Prasanna for putting this together. I agree with >> your comments in that Heketi is the high level abstraction API and it should >> have >> an API similar of what is described by Prasanna. >> >> I definitely do not think any File Api should be available in Heketi, >> because that is an implementation of the Block API. The Heketi API should >> be similar to something like OpenStack Cinder. >> >> I think that the actual management of the Volumes used for Block storage >> and the files in them should be all managed by Heketi. How they are >> actually created is still to be determined, but we could have Heketi >> create them, or have helper programs do that. > > Maybe a tool like qemu-img? If whatever iscsi service understand the > format (at the very least 'raw'), you could get functionality like > snapshots pretty simple. Niels, This is brilliant and subset of the Idea falls in one among my thoughts, only concern is about building dependencies of qemu with Heketi. But at an advantage of easy and cool snapshots solution. -- Prasanna > > Niels > > >> We also need to document the exact workflow to enable a file in >> a Gluster volume to be exposed as a block device. This will help >> determine where the creation of the file could take place. >> >> We can capture our decisions from these discussions in the >> following page: >> >> https://github.com/heketi/heketi/wiki/Proposed-Changes >> >> - Luis >> >> >> - Original Message - >> From: "Humble Chirammal" >> To: "Raghavendra Talur" >> Cc: "Prasanna Kalever" , "gluster-devel" >> , "Stephen Watt" , "Luis Pabon" >> , "Michael Adam" , "Ramakrishna >> Yekulla" , "Mohamed Ashiq Liyazudeen" >> >> Sent: Tuesday, September 13, 2016 2:23:39 AM >> Subject: Re: [Gluster-devel] [Heketi] Block store related API design >> discussion >> >> >> >> >> >> ----- Original Message - >> | From: "Raghavendra Talur" >> | To: "Prasanna Kalever" >> | Cc: "gluster-devel" , "Stephen Watt" >> , "Luis Pabon" , >> | "Michael Adam" , "Humble Chirammal" >> , "Ramakrishna Yekulla" >> | , "Mohamed Ashiq Liyazudeen" >> | Sent: Tuesday, September 13, 2016 11:08:44 AM >> | Subject: Re: [Gluster-devel] [Heketi] Block store related API design >> discussion >> | >> | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever >> | wrote: >> | >> | > Hi all, >> | > >> | > This mail is open for discussion on gluster block store integration with >> | > heketi and its REST API interface design constraints. >> | > >> | > >> | > ___ Volume Request ... >> | > | >> | > | >> | > PVC claim -> Heketi --->| >> | > | >> | > | >> | > | >> | > | >> | > |__ BlockCreate >> | > | | >> | > | |__ BlockInfo >> | > | | >> | > |___ Block Request (APIS)-> |__ BlockResize >> | > | >> | > |__ BlockList >> | > | >> | > |__ BlockDelete >> | > >> | > Heketi will have block API and volume API, when user submit a Persistent >> | > volume claim, Kubernetes provisioner based on the storage class(from PVC) >> | > talks to heketi for storage, heketi intern calls block or volume API's >> | > based on request. >> | > >> | >> | This is probably wrong. It won't be Heketi calling block or volume APIs. It >> | would be Kubernetes calling block or volume API *of* Heketi. >> | >> | >> | > With my limited understanding, heketi currently creates clusters from >> | > provided nodes, creates volumes and handover them to the u
Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Mon, Sep 19, 2016 at 9:05 AM, Luis Pabón wrote: > Hi Prasanna, > I started the wiki page with the documentation on the API. There > still needs to be more information added, and we still need to work > on the workflow, but at least it is a start. > > Please take a look at the wiki: > > https://github.com/heketi/heketi/wiki/Proposed-API:-Block-Storage This is cool Luis, we will append the workflow and other info needed there based on our future discussions. Thanks for keeping this in a brilliant way, you made it simple and easy to append info over the doc. -- Prasanna > > - Luis > > - Original Message - > From: "Luis Pabón" > To: "Humble Chirammal" > Cc: "gluster-devel" , "Stephen Watt" > , "Ramakrishna Yekulla" > Sent: Tuesday, September 13, 2016 12:06:00 PM > Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > > Very good points. Thanks Prasanna for putting this together. I agree with > your comments in that Heketi is the high level abstraction API and it should > have > an API similar of what is described by Prasanna. > > I definitely do not think any File Api should be available in Heketi, > because that is an implementation of the Block API. The Heketi API should > be similar to something like OpenStack Cinder. > > I think that the actual management of the Volumes used for Block storage > and the files in them should be all managed by Heketi. How they are > actually created is still to be determined, but we could have Heketi > create them, or have helper programs do that. > > We also need to document the exact workflow to enable a file in > a Gluster volume to be exposed as a block device. This will help > determine where the creation of the file could take place. > > We can capture our decisions from these discussions in the > following page: > > https://github.com/heketi/heketi/wiki/Proposed-Changes > > - Luis > > > - Original Message - > From: "Humble Chirammal" > To: "Raghavendra Talur" > Cc: "Prasanna Kalever" , "gluster-devel" > , "Stephen Watt" , "Luis Pabon" > , "Michael Adam" , "Ramakrishna Yekulla" > , "Mohamed Ashiq Liyazudeen" > Sent: Tuesday, September 13, 2016 2:23:39 AM > Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > > > > > > - Original Message - > | From: "Raghavendra Talur" > | To: "Prasanna Kalever" > | Cc: "gluster-devel" , "Stephen Watt" > , "Luis Pabon" , > | "Michael Adam" , "Humble Chirammal" > , "Ramakrishna Yekulla" > | , "Mohamed Ashiq Liyazudeen" > | Sent: Tuesday, September 13, 2016 11:08:44 AM > | Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > | > | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever > | wrote: > | > | > Hi all, > | > > | > This mail is open for discussion on gluster block store integration with > | > heketi and its REST API interface design constraints. > | > > | > > | > ___ Volume Request ... > | > | > | > | > | > PVC claim -> Heketi --->| > | > | > | > | > | > | > | > | > | > |__ BlockCreate > | > | | > | > | |__ BlockInfo > | > | | > | > |___ Block Request (APIS)-> |__ BlockResize > | > | > | > |__ BlockList > | > | > | > |__ BlockDelete > | > > | > Heketi will have block API and volume API, when user submit a Persistent > | > volume claim, Kubernetes provisioner based on the storage class(from PVC) > | > talks to heketi for storage, heketi intern calls block or volume API's > | > based on request. > | > > | > | This is probably wrong. It won't be Heketi calling block or volume APIs. It > | would be Kubernetes calling block or volume API *of* Heketi. > | > | > | > With my limited understanding, heketi currently creates clusters from >
Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: > Very good points. Thanks Prasanna for putting this together. I agree with > your comments in that Heketi is the high level abstraction API and it should > have > an API similar of what is described by Prasanna. > > I definitely do not think any File Api should be available in Heketi, > because that is an implementation of the Block API. The Heketi API should > be similar to something like OpenStack Cinder. > > I think that the actual management of the Volumes used for Block storage > and the files in them should be all managed by Heketi. How they are > actually created is still to be determined, but we could have Heketi > create them, or have helper programs do that. Maybe a tool like qemu-img? If whatever iscsi service understand the format (at the very least 'raw'), you could get functionality like snapshots pretty simple. Niels > We also need to document the exact workflow to enable a file in > a Gluster volume to be exposed as a block device. This will help > determine where the creation of the file could take place. > > We can capture our decisions from these discussions in the > following page: > > https://github.com/heketi/heketi/wiki/Proposed-Changes > > - Luis > > > - Original Message - > From: "Humble Chirammal" > To: "Raghavendra Talur" > Cc: "Prasanna Kalever" , "gluster-devel" > , "Stephen Watt" , "Luis Pabon" > , "Michael Adam" , "Ramakrishna Yekulla" > , "Mohamed Ashiq Liyazudeen" > Sent: Tuesday, September 13, 2016 2:23:39 AM > Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > > > > > > - Original Message - > | From: "Raghavendra Talur" > | To: "Prasanna Kalever" > | Cc: "gluster-devel" , "Stephen Watt" > , "Luis Pabon" , > | "Michael Adam" , "Humble Chirammal" > , "Ramakrishna Yekulla" > | , "Mohamed Ashiq Liyazudeen" > | Sent: Tuesday, September 13, 2016 11:08:44 AM > | Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > | > | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever > | wrote: > | > | > Hi all, > | > > | > This mail is open for discussion on gluster block store integration with > | > heketi and its REST API interface design constraints. > | > > | > > | > ___ Volume Request ... > | > | > | > | > | > PVC claim -> Heketi --->| > | > | > | > | > | > | > | > | > | > |__ BlockCreate > | > | | > | > | |__ BlockInfo > | > | | > | > |___ Block Request (APIS)-> |__ BlockResize > | > | > | > |__ BlockList > | > | > | > |__ BlockDelete > | > > | > Heketi will have block API and volume API, when user submit a Persistent > | > volume claim, Kubernetes provisioner based on the storage class(from PVC) > | > talks to heketi for storage, heketi intern calls block or volume API's > | > based on request. > | > > | > | This is probably wrong. It won't be Heketi calling block or volume APIs. It > | would be Kubernetes calling block or volume API *of* Heketi. > | > | > | > With my limited understanding, heketi currently creates clusters from > | > provided nodes, creates volumes and handover them to the user. > | > For block related API's, it has to deal with files right ? > | > > | > Here is how block API's look like in short- > | > Create: heketi has to create file in the volume and export it as a iscsi > | > target device and hand it over to user. > | > Info: show block stores information across all the clusters, connection > | > info, size etc. > | > resize: resize the file in the volume, refresh connections from initiator > | > side > | > List: List the connections > | > Delete: logout the connections and delete the file in the gluster volume > | > > | > Couple of qu
Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Tue, Sep 13, 2016 at 12:10 PM, Stephen Watt wrote: > > Also, some important requirements to figure out/think about are: > > - How are you managing locking a block device against a container (or a > host?) > - Will your implementation work with OpenShift volume security for block > devices (FSGroups + Recursive chown, chmod and SELinux labeling) > > If these aren't already figured out, would it be possible to create separate > cards in your trello board so we can track the progress on the resolution of > these two topics? > Tracking through Trello cards [1] and [2]. Thanks, Vijay [1] https://trello.com/c/LvfuP1cB/4-read-write-once-limit-a-block-device-to-a-single-container [2] https://trello.com/c/Ne9NeU2y/43-review-openshift-volume-security-for-block-devices > On Tue, Sep 13, 2016 at 11:06 AM, Luis Pabón wrote: >> >> Very good points. Thanks Prasanna for putting this together. I agree >> with >> your comments in that Heketi is the high level abstraction API and it >> should have >> an API similar of what is described by Prasanna. >> >> I definitely do not think any File Api should be available in Heketi, >> because that is an implementation of the Block API. The Heketi API should >> be similar to something like OpenStack Cinder. >> >> I think that the actual management of the Volumes used for Block storage >> and the files in them should be all managed by Heketi. How they are >> actually created is still to be determined, but we could have Heketi >> create them, or have helper programs do that. >> >> We also need to document the exact workflow to enable a file in >> a Gluster volume to be exposed as a block device. This will help >> determine where the creation of the file could take place. >> >> We can capture our decisions from these discussions in the >> following page: >> >> https://github.com/heketi/heketi/wiki/Proposed-Changes >> >> - Luis >> >> >> - Original Message - >> From: "Humble Chirammal" >> To: "Raghavendra Talur" >> Cc: "Prasanna Kalever" , "gluster-devel" >> , "Stephen Watt" , "Luis Pabon" >> , "Michael Adam" , "Ramakrishna >> Yekulla" , "Mohamed Ashiq Liyazudeen" >> >> Sent: Tuesday, September 13, 2016 2:23:39 AM >> Subject: Re: [Gluster-devel] [Heketi] Block store related API design >> discussion >> >> >> >> >> >> - Original Message - >> | From: "Raghavendra Talur" >> | To: "Prasanna Kalever" >> | Cc: "gluster-devel" , "Stephen Watt" >> , "Luis Pabon" , >> | "Michael Adam" , "Humble Chirammal" >> , "Ramakrishna Yekulla" >> | , "Mohamed Ashiq Liyazudeen" >> | Sent: Tuesday, September 13, 2016 11:08:44 AM >> | Subject: Re: [Gluster-devel] [Heketi] Block store related API design >> discussion >> | >> | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever >> | wrote: >> | >> | > Hi all, >> | > >> | > This mail is open for discussion on gluster block store integration >> with >> | > heketi and its REST API interface design constraints. >> | > >> | > >> | > ___ Volume Request ... >> | > | >> | > | >> | > PVC claim -> Heketi --->| >> | > | >> | > | >> | > | >> | > | >> | > |__ BlockCreate >> | > | | >> | > | |__ BlockInfo >> | > | | >> | > |___ Block Request (APIS)-> |__ BlockResize >> | > | >> | > |__ BlockList >> | > | >> | > |__ BlockDelete >> | > >> | > Heketi will have block API and volume API, when user submit a >> Persistent >> | > volume claim, Kubernetes provisioner based on the storage class(from >> PVC) >> | > talks to heketi for storage, heketi intern calls block or volume API's >> | > based on request
Re: [Gluster-devel] [Heketi] Block store related API design discussion
Hi Prasanna, I started the wiki page with the documentation on the API. There still needs to be more information added, and we still need to work on the workflow, but at least it is a start. Please take a look at the wiki: https://github.com/heketi/heketi/wiki/Proposed-API:-Block-Storage - Luis - Original Message - From: "Luis Pabón" To: "Humble Chirammal" Cc: "gluster-devel" , "Stephen Watt" , "Ramakrishna Yekulla" Sent: Tuesday, September 13, 2016 12:06:00 PM Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion Very good points. Thanks Prasanna for putting this together. I agree with your comments in that Heketi is the high level abstraction API and it should have an API similar of what is described by Prasanna. I definitely do not think any File Api should be available in Heketi, because that is an implementation of the Block API. The Heketi API should be similar to something like OpenStack Cinder. I think that the actual management of the Volumes used for Block storage and the files in them should be all managed by Heketi. How they are actually created is still to be determined, but we could have Heketi create them, or have helper programs do that. We also need to document the exact workflow to enable a file in a Gluster volume to be exposed as a block device. This will help determine where the creation of the file could take place. We can capture our decisions from these discussions in the following page: https://github.com/heketi/heketi/wiki/Proposed-Changes - Luis - Original Message - From: "Humble Chirammal" To: "Raghavendra Talur" Cc: "Prasanna Kalever" , "gluster-devel" , "Stephen Watt" , "Luis Pabon" , "Michael Adam" , "Ramakrishna Yekulla" , "Mohamed Ashiq Liyazudeen" Sent: Tuesday, September 13, 2016 2:23:39 AM Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion - Original Message - | From: "Raghavendra Talur" | To: "Prasanna Kalever" | Cc: "gluster-devel" , "Stephen Watt" , "Luis Pabon" , | "Michael Adam" , "Humble Chirammal" , "Ramakrishna Yekulla" | , "Mohamed Ashiq Liyazudeen" | Sent: Tuesday, September 13, 2016 11:08:44 AM | Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion | | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever | wrote: | | > Hi all, | > | > This mail is open for discussion on gluster block store integration with | > heketi and its REST API interface design constraints. | > | > | > ___ Volume Request ... | > | | > | | > PVC claim -> Heketi --->| | > | | > | | > | | > | | > |__ BlockCreate | > | | | > | |__ BlockInfo | > | | | > |___ Block Request (APIS)-> |__ BlockResize | > | | > |__ BlockList | > | | > |__ BlockDelete | > | > Heketi will have block API and volume API, when user submit a Persistent | > volume claim, Kubernetes provisioner based on the storage class(from PVC) | > talks to heketi for storage, heketi intern calls block or volume API's | > based on request. | > | | This is probably wrong. It won't be Heketi calling block or volume APIs. It | would be Kubernetes calling block or volume API *of* Heketi. | | | > With my limited understanding, heketi currently creates clusters from | > provided nodes, creates volumes and handover them to the user. | > For block related API's, it has to deal with files right ? | > | > Here is how block API's look like in short- | > Create: heketi has to create file in the volume and export it as a iscsi | > target device and hand it over to user. | > Info: show block stores information across all the clusters, connection | > info, size etc. | > resize: resize the file in the volume, refresh connections from initiator | > side | > List: List the connections | > Delete: logout the connections and delete the file in the gluster volume | > | > Couple of questions: | > 1. Should Block API have sub API's such as FileCreate, FileList, | > FileResize, File delete and etc then get it used in B
Re: [Gluster-devel] [Heketi] Block store related API design discussion
+ rhs-containers list Also, some important requirements to figure out/think about are: - How are you managing locking a block device against a container (or a host?) - Will your implementation work with OpenShift volume security for block devices (FSGroups + Recursive chown, chmod and SELinux labeling) If these aren't already figured out, would it be possible to create separate cards in your trello board so we can track the progress on the resolution of these two topics? On Tue, Sep 13, 2016 at 11:06 AM, Luis Pabón wrote: > Very good points. Thanks Prasanna for putting this together. I agree with > your comments in that Heketi is the high level abstraction API and it > should have > an API similar of what is described by Prasanna. > > I definitely do not think any File Api should be available in Heketi, > because that is an implementation of the Block API. The Heketi API should > be similar to something like OpenStack Cinder. > > I think that the actual management of the Volumes used for Block storage > and the files in them should be all managed by Heketi. How they are > actually created is still to be determined, but we could have Heketi > create them, or have helper programs do that. > > We also need to document the exact workflow to enable a file in > a Gluster volume to be exposed as a block device. This will help > determine where the creation of the file could take place. > > We can capture our decisions from these discussions in the > following page: > > https://github.com/heketi/heketi/wiki/Proposed-Changes > > - Luis > > > - Original Message - > From: "Humble Chirammal" > To: "Raghavendra Talur" > Cc: "Prasanna Kalever" , "gluster-devel" < > gluster-devel@gluster.org>, "Stephen Watt" , "Luis > Pabon" , "Michael Adam" , > "Ramakrishna Yekulla" , "Mohamed Ashiq Liyazudeen" < > mliya...@redhat.com> > Sent: Tuesday, September 13, 2016 2:23:39 AM > Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > > > > > > - Original Message - > | From: "Raghavendra Talur" > | To: "Prasanna Kalever" > | Cc: "gluster-devel" , "Stephen Watt" < > sw...@redhat.com>, "Luis Pabon" , > | "Michael Adam" , "Humble Chirammal" < > hchir...@redhat.com>, "Ramakrishna Yekulla" > | , "Mohamed Ashiq Liyazudeen" > | Sent: Tuesday, September 13, 2016 11:08:44 AM > | Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > | > | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever > | wrote: > | > | > Hi all, > | > > | > This mail is open for discussion on gluster block store integration > with > | > heketi and its REST API interface design constraints. > | > > | > > | > ___ Volume Request ... > | > | > | > | > | > PVC claim -> Heketi --->| > | > | > | > | > | > | > | > | > | > |__ BlockCreate > | > | | > | > | |__ BlockInfo > | > | | > | > |___ Block Request (APIS)-> |__ BlockResize > | > | > | > |__ BlockList > | > | > | > |__ BlockDelete > | > > | > Heketi will have block API and volume API, when user submit a > Persistent > | > volume claim, Kubernetes provisioner based on the storage class(from > PVC) > | > talks to heketi for storage, heketi intern calls block or volume API's > | > based on request. > | > > | > | This is probably wrong. It won't be Heketi calling block or volume APIs. > It > | would be Kubernetes calling block or volume API *of* Heketi. > | > | > | > With my limited understanding, heketi currently creates clusters from > | > provided nodes, creates volumes and handover them to the user. > | > For block related API's, it has to deal with files right ? > | > > | > Here is how block API's look like in short- > | > Create: heketi has to create file in the volume and export it as a > iscsi > | &
Re: [Gluster-devel] [Heketi] Block store related API design discussion
Hi Steve, Good questions. We still need to investigate the security concerns around block storage. - Luis - Original Message - From: "Stephen Watt" To: "Luis Pabón" Cc: "Humble Chirammal" , "Raghavendra Talur" , "Prasanna Kalever" , "gluster-devel" , "Michael Adam" , "Ramakrishna Yekulla" , "Mohamed Ashiq Liyazudeen" , "Engineering discussions on containers & RHS" Sent: Tuesday, September 13, 2016 12:10:54 PM Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion + rhs-containers list Also, some important requirements to figure out/think about are: - How are you managing locking a block device against a container (or a host?) - Will your implementation work with OpenShift volume security for block devices (FSGroups + Recursive chown, chmod and SELinux labeling) If these aren't already figured out, would it be possible to create separate cards in your trello board so we can track the progress on the resolution of these two topics? On Tue, Sep 13, 2016 at 11:06 AM, Luis Pabón wrote: > Very good points. Thanks Prasanna for putting this together. I agree with > your comments in that Heketi is the high level abstraction API and it > should have > an API similar of what is described by Prasanna. > > I definitely do not think any File Api should be available in Heketi, > because that is an implementation of the Block API. The Heketi API should > be similar to something like OpenStack Cinder. > > I think that the actual management of the Volumes used for Block storage > and the files in them should be all managed by Heketi. How they are > actually created is still to be determined, but we could have Heketi > create them, or have helper programs do that. > > We also need to document the exact workflow to enable a file in > a Gluster volume to be exposed as a block device. This will help > determine where the creation of the file could take place. > > We can capture our decisions from these discussions in the > following page: > > https://github.com/heketi/heketi/wiki/Proposed-Changes > > - Luis > > > - Original Message - > From: "Humble Chirammal" > To: "Raghavendra Talur" > Cc: "Prasanna Kalever" , "gluster-devel" < > gluster-devel@gluster.org>, "Stephen Watt" , "Luis > Pabon" , "Michael Adam" , > "Ramakrishna Yekulla" , "Mohamed Ashiq Liyazudeen" < > mliya...@redhat.com> > Sent: Tuesday, September 13, 2016 2:23:39 AM > Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > > > > > > - Original Message ----- > | From: "Raghavendra Talur" > | To: "Prasanna Kalever" > | Cc: "gluster-devel" , "Stephen Watt" < > sw...@redhat.com>, "Luis Pabon" , > | "Michael Adam" , "Humble Chirammal" < > hchir...@redhat.com>, "Ramakrishna Yekulla" > | , "Mohamed Ashiq Liyazudeen" > | Sent: Tuesday, September 13, 2016 11:08:44 AM > | Subject: Re: [Gluster-devel] [Heketi] Block store related API design > discussion > | > | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever > | wrote: > | > | > Hi all, > | > > | > This mail is open for discussion on gluster block store integration > with > | > heketi and its REST API interface design constraints. > | > > | > > | > ___ Volume Request ... > | > | > | > | > | > PVC claim -> Heketi --->| > | > | > | > | > | > | > | > | > | > |__ BlockCreate > | > | | > | > | |__ BlockInfo > | > | | > | > |___ Block Request (APIS)-> |__ BlockResize > | > | > | > |__ BlockList > | > | > | > |__ BlockDelete > | > > | > Heketi will have block API and volume API, when user submit a > Persistent > | > volume claim, Kubernetes provisioner based on the storage class(from > PVC) > | > talks to heketi for storage, heketi intern calls block or volume API's > |
Re: [Gluster-devel] [Heketi] Block store related API design discussion
Very good points. Thanks Prasanna for putting this together. I agree with your comments in that Heketi is the high level abstraction API and it should have an API similar of what is described by Prasanna. I definitely do not think any File Api should be available in Heketi, because that is an implementation of the Block API. The Heketi API should be similar to something like OpenStack Cinder. I think that the actual management of the Volumes used for Block storage and the files in them should be all managed by Heketi. How they are actually created is still to be determined, but we could have Heketi create them, or have helper programs do that. We also need to document the exact workflow to enable a file in a Gluster volume to be exposed as a block device. This will help determine where the creation of the file could take place. We can capture our decisions from these discussions in the following page: https://github.com/heketi/heketi/wiki/Proposed-Changes - Luis - Original Message - From: "Humble Chirammal" To: "Raghavendra Talur" Cc: "Prasanna Kalever" , "gluster-devel" , "Stephen Watt" , "Luis Pabon" , "Michael Adam" , "Ramakrishna Yekulla" , "Mohamed Ashiq Liyazudeen" Sent: Tuesday, September 13, 2016 2:23:39 AM Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion - Original Message - | From: "Raghavendra Talur" | To: "Prasanna Kalever" | Cc: "gluster-devel" , "Stephen Watt" , "Luis Pabon" , | "Michael Adam" , "Humble Chirammal" , "Ramakrishna Yekulla" | , "Mohamed Ashiq Liyazudeen" | Sent: Tuesday, September 13, 2016 11:08:44 AM | Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion | | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever | wrote: | | > Hi all, | > | > This mail is open for discussion on gluster block store integration with | > heketi and its REST API interface design constraints. | > | > | > ___ Volume Request ... | > | | > | | > PVC claim -> Heketi --->| | > | | > | | > | | > | | > |__ BlockCreate | > | | | > | |__ BlockInfo | > | | | > |___ Block Request (APIS)-> |__ BlockResize | > | | > |__ BlockList | > | | > |__ BlockDelete | > | > Heketi will have block API and volume API, when user submit a Persistent | > volume claim, Kubernetes provisioner based on the storage class(from PVC) | > talks to heketi for storage, heketi intern calls block or volume API's | > based on request. | > | | This is probably wrong. It won't be Heketi calling block or volume APIs. It | would be Kubernetes calling block or volume API *of* Heketi. | | | > With my limited understanding, heketi currently creates clusters from | > provided nodes, creates volumes and handover them to the user. | > For block related API's, it has to deal with files right ? | > | > Here is how block API's look like in short- | > Create: heketi has to create file in the volume and export it as a iscsi | > target device and hand it over to user. | > Info: show block stores information across all the clusters, connection | > info, size etc. | > resize: resize the file in the volume, refresh connections from initiator | > side | > List: List the connections | > Delete: logout the connections and delete the file in the gluster volume | > | > Couple of questions: | > 1. Should Block API have sub API's such as FileCreate, FileList, | > FileResize, File delete and etc then get it used in Block API as they | > mostly deal with files. | > | | IMO, Heketi should not expose any File related API. It should only have | APIs to service request for block devices; how the block devices are | created and modified is an implementation detail. | | | > 2. How do we create the actual file in the volume, meaning using FUSE | > mount (which may involve an extra process running) or gfapi, again if gfapi | > should we go with c API's, python bindings or go bindings ? | > | 3. Should we get targetcli related (LUN exporting) setup done from heketi | > or do we seek help from gdeploy for
Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever wrote: > Hi all, > > This mail is open for discussion on gluster block store integration with > heketi and its REST API interface design constraints. > > > ___ Volume Request ... > | > | > PVC claim -> Heketi --->| > | > | > | > | > |__ BlockCreate > | | > | |__ BlockInfo > | | > |___ Block Request (APIS)-> |__ BlockResize > | > |__ BlockList > | > |__ BlockDelete > > Heketi will have block API and volume API, when user submit a Persistent > volume claim, Kubernetes provisioner based on the storage class(from PVC) > talks to heketi for storage, heketi intern calls block or volume API's > based on request. > This is probably wrong. It won't be Heketi calling block or volume APIs. It would be Kubernetes calling block or volume API *of* Heketi. > With my limited understanding, heketi currently creates clusters from > provided nodes, creates volumes and handover them to the user. > For block related API's, it has to deal with files right ? > > Here is how block API's look like in short- > Create: heketi has to create file in the volume and export it as a iscsi > target device and hand it over to user. > Info: show block stores information across all the clusters, connection > info, size etc. > resize: resize the file in the volume, refresh connections from initiator > side > List: List the connections > Delete: logout the connections and delete the file in the gluster volume > > Couple of questions: > 1. Should Block API have sub API's such as FileCreate, FileList, > FileResize, File delete and etc then get it used in Block API as they > mostly deal with files. > IMO, Heketi should not expose any File related API. It should only have APIs to service request for block devices; how the block devices are created and modified is an implementation detail. > 2. How do we create the actual file in the volume, meaning using FUSE > mount (which may involve an extra process running) or gfapi, again if gfapi > should we go with c API's, python bindings or go bindings ? > 3. Should we get targetcli related (LUN exporting) setup done from heketi > or do we seek help from gdeploy for this ? > I would prefer to either have it in Heketi or in Kubernetes. If the API in Heketi promises just the creation of block device, then the rest of the implementation should be in Kubernetes(the export part). If the API in Heketi promises create and export both, I would say Heketi should have the implementation within itself. > > > Thoughts? > > Note: nothing is fixed as put in this mail, its all just part of initial > discussions. > > > Cheers, > -- > Prasanna > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel