When you say, "wire up the lun directly to the vm," do you mean
circumventing the hypervisor? I didn't think we could do that in CS.
OpenStack, on the other hand, always circumvents the hypervisor, as far as
I know.
On Fri, Sep 13, 2013 at 7:40 PM, Marcus Sorensen <shadow...@gmail.com>wrote:
> Better to wire up the lun directly to the vm unless there is a good reason
> not to.
> On Sep 13, 2013 7:40 PM, "Marcus Sorensen" <shadow...@gmail.com> wrote:
>
>> You could do that, but as mentioned I think its a mistake to go to the
>> trouble of creating a 1:1 mapping of CS volumes to luns and then putting a
>> filesystem on it, mounting it, and then putting a QCOW2 or even RAW disk
>> image on that filesystem. You'll lose a lot of iops along the way, and have
>> more overhead with the filesystem and its journaling, etc.
>> On Sep 13, 2013 7:33 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com>
>> wrote:
>>
>>> Ah, OK, I didn't know that was such new ground in KVM with CS.
>>>
>>> So, the way people use our SAN with KVM and CS today is by selecting
>>> SharedMountPoint and specifying the location of the share.
>>>
>>> They can set up their share using Open iSCSI by discovering their iSCSI
>>> target, logging in to it, then mounting it somewhere on their file system.
>>>
>>> Would it make sense for me to just do that discovery, logging in, and
>>> mounting behind the scenes for them and letting the current code manage the
>>> rest as it currently does?
>>>
>>>
>>> On Fri, Sep 13, 2013 at 7:27 PM, Marcus Sorensen <shadow...@gmail.com>wrote:
>>>
>>>> Oh, hypervisor snapshots are a bit different. I need to catch up on the
>>>> work done in KVM, but this is basically just disk snapshots + memory dump.
>>>> I still think disk snapshots would preferably be handled by the SAN, and
>>>> then memory dumps can go to secondary storage or something else. This is
>>>> relatively new ground with CS and KVM, so we will want to see how others
>>>> are planning theirs.
>>>> On Sep 13, 2013 7:20 PM, "Marcus Sorensen" <shadow...@gmail.com>
>>>> wrote:
>>>>
>>>>> Let me back up and say I don't think you'd use a vdi style on an iscsi
>>>>> lun. I think you'd want to treat it as a RAW format. Otherwise you're
>>>>> putting a filesystem on your lun, mounting it, creating a QCOW2 disk
>>>>> image,
>>>>> and that seems unnecessary and a performance killer.
>>>>>
>>>>> So probably attaching the raw iscsi lun as a disk to the VM, and
>>>>> handling snapshots on the San side via the storage plugin is best. My
>>>>> impression from the storage plugin refactor was that there was a snapshot
>>>>> service that would allow the San to handle snapshots.
>>>>> On Sep 13, 2013 7:15 PM, "Marcus Sorensen" <shadow...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Ideally volume snapshots can be handled by the SAN back end, if the
>>>>>> SAN supports it. The cloudstack mgmt server could call your plugin for
>>>>>> volume snapshot and it would be hypervisor agnostic. As far as space,
>>>>>> that
>>>>>> would depend on how your SAN handles it. With ours, we carve out luns
>>>>>> from
>>>>>> a pool, and the snapshot spave comes from the pool and is independent of
>>>>>> the LUN size the host sees.
>>>>>> On Sep 13, 2013 7:10 PM, "Mike Tutkowski" <
>>>>>> mike.tutkow...@solidfire.com> wrote:
>>>>>>
>>>>>>> Hey Marcus,
>>>>>>>
>>>>>>> I wonder if the iSCSI storage pool type for libvirt won't work when
>>>>>>> you take into consideration hypervisor snapshots?
>>>>>>>
>>>>>>> On XenServer, when you take a hypervisor snapshot, the VDI for the
>>>>>>> snapshot is placed on the same storage repository as the volume is on.
>>>>>>>
>>>>>>> Same idea for VMware, I believe.
>>>>>>>
>>>>>>> So, what would happen in my case (let's say for XenServer and VMware
>>>>>>> for 4.3 because I don't support hypervisor snapshots in 4.2) is I'd
>>>>>>> make an
>>>>>>> iSCSI target that is larger than what the user requested for the
>>>>>>> CloudStack
>>>>>>> volume (which is fine because our SAN thinly provisions volumes, so the
>>>>>>> space is not actually used unless it needs to be). The CloudStack volume
>>>>>>> would be the only "object" on the SAN volume until a hypervisor
>>>>>>> snapshot is
>>>>>>> taken. This snapshot would also reside on the SAN volume.
>>>>>>>
>>>>>>> If this is also how KVM behaves and there is no creation of LUNs
>>>>>>> within an iSCSI target from libvirt (which, even if there were support
>>>>>>> for
>>>>>>> this, our SAN currently only allows one LUN per iSCSI target), then I
>>>>>>> don't
>>>>>>> see how using this model will work.
>>>>>>>
>>>>>>> Perhaps I will have to go enhance the current way this works with
>>>>>>> DIR?
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 13, 2013 at 6:28 PM, Mike Tutkowski <
>>>>>>> mike.tutkow...@solidfire.com> wrote:
>>>>>>>
>>>>>>>> That appears to be the way it's used for iSCSI access today.
>>>>>>>>
>>>>>>>> I suppose I could go that route, too, but I might as well leverage
>>>>>>>> what libvirt has for iSCSI instead.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Sep 13, 2013 at 6:26 PM, Marcus Sorensen <
>>>>>>>> shadow...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> To your question about SharedMountPoint, I believe it just acts
>>>>>>>>> like a
>>>>>>>>> 'DIR' storage type or something similar to that. The end-user is
>>>>>>>>> responsible for mounting a file system that all KVM hosts can
>>>>>>>>> access,
>>>>>>>>> and CloudStack is oblivious to what is providing the storage. It
>>>>>>>>> could
>>>>>>>>> be NFS, or OCFS2, or some other clustered filesystem, cloudstack
>>>>>>>>> just
>>>>>>>>> knows that the provided directory path has VM images.
>>>>>>>>>
>>>>>>>>> On Fri, Sep 13, 2013 at 6:23 PM, Marcus Sorensen <
>>>>>>>>> shadow...@gmail.com> wrote:
>>>>>>>>> > Oh yes, you can use NFS, LVM, and iSCSI all at the same time.
>>>>>>>>> > Multiples, in fact.
>>>>>>>>> >
>>>>>>>>> > On Fri, Sep 13, 2013 at 6:19 PM, Mike Tutkowski
>>>>>>>>> > <mike.tutkow...@solidfire.com> wrote:
>>>>>>>>> >> Looks like you can have multiple storage pools:
>>>>>>>>> >>
>>>>>>>>> >> mtutkowski@ubuntu:~$ virsh pool-list
>>>>>>>>> >> Name State Autostart
>>>>>>>>> >> -----------------------------------------
>>>>>>>>> >> default active yes
>>>>>>>>> >> iSCSI active no
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> On Fri, Sep 13, 2013 at 6:12 PM, Mike Tutkowski
>>>>>>>>> >> <mike.tutkow...@solidfire.com> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> Reading through the docs you pointed out.
>>>>>>>>> >>>
>>>>>>>>> >>> I see what you're saying now.
>>>>>>>>> >>>
>>>>>>>>> >>> You can create an iSCSI (libvirt) storage pool based on an
>>>>>>>>> iSCSI target.
>>>>>>>>> >>>
>>>>>>>>> >>> In my case, the iSCSI target would only have one LUN, so there
>>>>>>>>> would only
>>>>>>>>> >>> be one iSCSI (libvirt) storage volume in the (libvirt) storage
>>>>>>>>> pool.
>>>>>>>>> >>>
>>>>>>>>> >>> As you say, my plug-in creates and destroys iSCSI targets/LUNs
>>>>>>>>> on the
>>>>>>>>> >>> SolidFire SAN, so it is not a problem that libvirt does not
>>>>>>>>> support
>>>>>>>>> >>> creating/deleting iSCSI targets/LUNs.
>>>>>>>>> >>>
>>>>>>>>> >>> It looks like I need to test this a bit to see if libvirt
>>>>>>>>> supports
>>>>>>>>> >>> multiple iSCSI storage pools (as you mentioned, since each one
>>>>>>>>> of its
>>>>>>>>> >>> storage pools would map to one of my iSCSI targets/LUNs).
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> On Fri, Sep 13, 2013 at 5:58 PM, Mike Tutkowski
>>>>>>>>> >>> <mike.tutkow...@solidfire.com> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> LibvirtStoragePoolDef has this type:
>>>>>>>>> >>>>
>>>>>>>>> >>>> public enum poolType {
>>>>>>>>> >>>>
>>>>>>>>> >>>> ISCSI("iscsi"), NETFS("netfs"), LOGICAL("logical"),
>>>>>>>>> DIR("dir"),
>>>>>>>>> >>>> RBD("rbd");
>>>>>>>>> >>>>
>>>>>>>>> >>>> String _poolType;
>>>>>>>>> >>>>
>>>>>>>>> >>>> poolType(String poolType) {
>>>>>>>>> >>>>
>>>>>>>>> >>>> _poolType = poolType;
>>>>>>>>> >>>>
>>>>>>>>> >>>> }
>>>>>>>>> >>>>
>>>>>>>>> >>>> @Override
>>>>>>>>> >>>>
>>>>>>>>> >>>> public String toString() {
>>>>>>>>> >>>>
>>>>>>>>> >>>> return _poolType;
>>>>>>>>> >>>>
>>>>>>>>> >>>> }
>>>>>>>>> >>>>
>>>>>>>>> >>>> }
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>> It doesn't look like the iSCSI type is currently being used,
>>>>>>>>> but I'm
>>>>>>>>> >>>> understanding more what you were getting at.
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>> Can you tell me for today (say, 4.2), when someone selects the
>>>>>>>>> >>>> SharedMountPoint option and uses it with iSCSI, is that the
>>>>>>>>> "netfs" option
>>>>>>>>> >>>> above or is that just for NFS?
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>> Thanks!
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Fri, Sep 13, 2013 at 5:50 PM, Marcus Sorensen <
>>>>>>>>> shadow...@gmail.com>
>>>>>>>>> >>>> wrote:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Take a look at this:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> http://libvirt.org/storage.html#StorageBackendISCSI
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> "Volumes must be pre-allocated on the iSCSI server, and
>>>>>>>>> cannot be
>>>>>>>>> >>>>> created via the libvirt APIs.", which I believe your plugin
>>>>>>>>> will take
>>>>>>>>> >>>>> care of. Libvirt just does the work of logging in and
>>>>>>>>> hooking it up to
>>>>>>>>> >>>>> the VM (I believe the Xen api does that work in the Xen
>>>>>>>>> stuff).
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> What I'm not sure about is whether this provides a 1:1
>>>>>>>>> mapping, or if
>>>>>>>>> >>>>> it just allows you to register 1 iscsi device as a pool. You
>>>>>>>>> may need
>>>>>>>>> >>>>> to write some test code or read up a bit more about this.
>>>>>>>>> Let us know.
>>>>>>>>> >>>>> If it doesn't, you may just have to write your own storage
>>>>>>>>> adaptor
>>>>>>>>> >>>>> rather than changing LibvirtStorageAdaptor.java. We can
>>>>>>>>> cross that
>>>>>>>>> >>>>> bridge when we get there.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> As far as interfacing with libvirt, see the java bindings
>>>>>>>>> doc.
>>>>>>>>> >>>>> http://libvirt.org/sources/java/javadoc/ Normally, you'll
>>>>>>>>> see a
>>>>>>>>> >>>>> connection object be made, then calls made to that 'conn'
>>>>>>>>> object. You
>>>>>>>>> >>>>> can look at the LibvirtStorageAdaptor to see how that is
>>>>>>>>> done for
>>>>>>>>> >>>>> other pool types, and maybe write some test java code to see
>>>>>>>>> if you
>>>>>>>>> >>>>> can interface with libvirt and register iscsi storage pools
>>>>>>>>> before you
>>>>>>>>> >>>>> get started.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Fri, Sep 13, 2013 at 5:31 PM, Mike Tutkowski
>>>>>>>>> >>>>> <mike.tutkow...@solidfire.com> wrote:
>>>>>>>>> >>>>> > So, Marcus, I need to investigate libvirt more, but you
>>>>>>>>> figure it
>>>>>>>>> >>>>> > supports
>>>>>>>>> >>>>> > connecting to/disconnecting from iSCSI targets, right?
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>> > On Fri, Sep 13, 2013 at 5:29 PM, Mike Tutkowski
>>>>>>>>> >>>>> > <mike.tutkow...@solidfire.com> wrote:
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> OK, thanks, Marcus
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> I am currently looking through some of the classes you
>>>>>>>>> pointed out
>>>>>>>>> >>>>> >> last
>>>>>>>>> >>>>> >> week or so.
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> On Fri, Sep 13, 2013 at 5:26 PM, Marcus Sorensen
>>>>>>>>> >>>>> >> <shadow...@gmail.com>
>>>>>>>>> >>>>> >> wrote:
>>>>>>>>> >>>>> >>>
>>>>>>>>> >>>>> >>> Yes, my guess is that you will need the iscsi initiator
>>>>>>>>> utilities
>>>>>>>>> >>>>> >>> installed. There should be standard packages for any
>>>>>>>>> distro. Then
>>>>>>>>> >>>>> >>> you'd call
>>>>>>>>> >>>>> >>> an agent storage adaptor to do the initiator login. See
>>>>>>>>> the info I
>>>>>>>>> >>>>> >>> sent
>>>>>>>>> >>>>> >>> previously about LibvirtStorageAdaptor.java and libvirt
>>>>>>>>> iscsi
>>>>>>>>> >>>>> >>> storage type
>>>>>>>>> >>>>> >>> to see if that fits your need.
>>>>>>>>> >>>>> >>>
>>>>>>>>> >>>>> >>> On Sep 13, 2013 4:55 PM, "Mike Tutkowski"
>>>>>>>>> >>>>> >>> <mike.tutkow...@solidfire.com>
>>>>>>>>> >>>>> >>> wrote:
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> Hi,
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> As you may remember, during the 4.2 release I developed
>>>>>>>>> a SolidFire
>>>>>>>>> >>>>> >>>> (storage) plug-in for CloudStack.
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> This plug-in was invoked by the storage framework at
>>>>>>>>> the necessary
>>>>>>>>> >>>>> >>>> times
>>>>>>>>> >>>>> >>>> so that I could dynamically create and delete volumes
>>>>>>>>> on the
>>>>>>>>> >>>>> >>>> SolidFire SAN
>>>>>>>>> >>>>> >>>> (among other activities).
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> This is necessary so I can establish a 1:1 mapping
>>>>>>>>> between a
>>>>>>>>> >>>>> >>>> CloudStack
>>>>>>>>> >>>>> >>>> volume and a SolidFire volume for QoS.
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> In the past, CloudStack always expected the admin to
>>>>>>>>> create large
>>>>>>>>> >>>>> >>>> volumes ahead of time and those volumes would likely
>>>>>>>>> house many
>>>>>>>>> >>>>> >>>> root and
>>>>>>>>> >>>>> >>>> data disks (which is not QoS friendly).
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> To make this 1:1 mapping scheme work, I needed to
>>>>>>>>> modify logic in
>>>>>>>>> >>>>> >>>> the
>>>>>>>>> >>>>> >>>> XenServer and VMware plug-ins so they could
>>>>>>>>> create/delete storage
>>>>>>>>> >>>>> >>>> repositories/datastores as needed.
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> For 4.3 I want to make this happen with KVM.
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> I'm coming up to speed with how this might work on KVM,
>>>>>>>>> but I'm
>>>>>>>>> >>>>> >>>> still
>>>>>>>>> >>>>> >>>> pretty new to KVM.
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> Does anyone familiar with KVM know how I will need to
>>>>>>>>> interact with
>>>>>>>>> >>>>> >>>> the
>>>>>>>>> >>>>> >>>> iSCSI target? For example, will I have to expect Open
>>>>>>>>> iSCSI will be
>>>>>>>>> >>>>> >>>> installed on the KVM host and use it for this to work?
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> Thanks for any suggestions,
>>>>>>>>> >>>>> >>>> Mike
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> --
>>>>>>>>> >>>>> >>>> Mike Tutkowski
>>>>>>>>> >>>>> >>>> Senior CloudStack Developer, SolidFire Inc.
>>>>>>>>> >>>>> >>>> e: mike.tutkow...@solidfire.com
>>>>>>>>> >>>>> >>>> o: 303.746.7302
>>>>>>>>> >>>>> >>>> Advancing the way the world uses the cloud™
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> --
>>>>>>>>> >>>>> >> Mike Tutkowski
>>>>>>>>> >>>>> >> Senior CloudStack Developer, SolidFire Inc.
>>>>>>>>> >>>>> >> e: mike.tutkow...@solidfire.com
>>>>>>>>> >>>>> >> o: 303.746.7302
>>>>>>>>> >>>>> >> Advancing the way the world uses the cloud™
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>> > --
>>>>>>>>> >>>>> > Mike Tutkowski
>>>>>>>>> >>>>> > Senior CloudStack Developer, SolidFire Inc.
>>>>>>>>> >>>>> > e: mike.tutkow...@solidfire.com
>>>>>>>>> >>>>> > o: 303.746.7302
>>>>>>>>> >>>>> > Advancing the way the world uses the cloud™
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>> --
>>>>>>>>> >>>> Mike Tutkowski
>>>>>>>>> >>>> Senior CloudStack Developer, SolidFire Inc.
>>>>>>>>> >>>> e: mike.tutkow...@solidfire.com
>>>>>>>>> >>>> o: 303.746.7302
>>>>>>>>> >>>> Advancing the way the world uses the cloud™
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> --
>>>>>>>>> >>> Mike Tutkowski
>>>>>>>>> >>> Senior CloudStack Developer, SolidFire Inc.
>>>>>>>>> >>> e: mike.tutkow...@solidfire.com
>>>>>>>>> >>> o: 303.746.7302
>>>>>>>>> >>> Advancing the way the world uses the cloud™
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> --
>>>>>>>>> >> Mike Tutkowski
>>>>>>>>> >> Senior CloudStack Developer, SolidFire Inc.
>>>>>>>>> >> e: mike.tutkow...@solidfire.com
>>>>>>>>> >> o: 303.746.7302
>>>>>>>>> >> Advancing the way the world uses the cloud™
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Mike Tutkowski*
>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>> e: mike.tutkow...@solidfire.com
>>>>>>>> o: 303.746.7302
>>>>>>>> Advancing the way the world uses the
>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>> *™*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Mike Tutkowski*
>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>> e: mike.tutkow...@solidfire.com
>>>>>>> o: 303.746.7302
>>>>>>> Advancing the way the world uses the
>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>> *™*
>>>>>>>
>>>>>>
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkow...@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the
>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>> *™*
>>>
>>
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*
