Re: [libvirt] [libguestfs] Options for hotplugging
On Mon, Aug 20, 2012 at 10:07:43PM +0100, Richard W.M. Jones wrote: In the first case (using /dev/sdX names), some magic already happens translating these to the real names underneath, but currently that magic is just /dev/sdX - /dev/vdX for the virtio case. Ah I didn't realize that you already do device name remapping between what the kernel shows what the API shows. This hidden drive doesn't appear in the API -- for example it is suppressed when we generate the result of guestfs_list_devices. Makes sense. (1) The raw libvirt option In this one we'd simply provide thin wrappers around virDomainAttachDevice and virDomainDetactDevice, and leave it up to the user to know what they're doing. The problem with this is the hidden appliance disk. We certainly don't want the user to accidentally detach that(!) It's also undesirable for there to be a hole in the naming scheme so that you'd have: /dev/sda - your normal drives /dev/sdb - [/dev/sdc # sorry, you can't use this, we won't tell you why] /dev/sdd - your first hotplugged device I think this scheme is flawed, because it exposes the user to the raw kernel device names (/dev/vda), which are not guaranteed to match the device names you expose in the API (/dev/sda). (2) The slots option In this option you'd have to use null devices to reserve the maximum number of drive slots that you're going to use in the libguestfs handle before launch. Then after launching you'd be allowed to hotplug only those slots. So for example: guestfs_add_drive (g, /dev/null); # reserves /dev/sda guestfs_add_drive (g, /dev/null); # reserves /dev/sdb guestfs_add_drive (g, /dev/null); # reserves /dev/sdc guestfs_launch (g); guestfs_hotplug (g, 1, /tmp/foo); # replaces index 1 == /dev/sdb guestfs_hotplug (g, 3, /tmp/foo); # error! Although ugly, in some ways this is quite attractive. It maps easily into guestfish scripts. You have contiguous device naming. You often know how many drives you'll need in advance, and if you don't then you can reserve up to max_disks-1. This feels rather unpleasant to me - I don't really consider it to be true hotplug if you have to plan it all in advance. (3) The serial numbers option This was Dan's suggestion. Hotplugged drives are known only by their serial number. ie. We hotplug them via libvirt using the serial/ field, and then they are accessed using /dev/disk/by-id/serial. This is tempting, but unfortunately it doesn't quite work in stock udev, because the actual name used is: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_SERIAL We could add a custom udev rule to get the path we wanted. Yep, you wouldn't even need to use any of the /dev/disk/by-XXX dirs at all - you could easily add /dev/guestfs/$SERIAL as a naming scheme if you wanted. (4) The rewriting device names option Since we already have the infrastructure to rewrite device names, we could do some complicated and hairy device name rewriting to make names appear continguous, even though there's an hidden appliance drive. This is my least favourite option, mainly because of the complexity, and complexity is bound to lead to bugs. Heh, based on the fact that you already have todo device name translation as described above, this feels like the best option to me. (5) Your idea here ... I think I'd do both (3) and (4), since I think (3) could be useful even outside the realm of hotplug. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [libguestfs] Options for hotplugging
On Tue, Aug 21, 2012 at 10:32:51AM +0100, Daniel P. Berrange wrote: On Mon, Aug 20, 2012 at 10:07:43PM +0100, Richard W.M. Jones wrote: In the first case (using /dev/sdX names), some magic already happens translating these to the real names underneath, but currently that magic is just /dev/sdX - /dev/vdX for the virtio case. Ah I didn't realize that you already do device name remapping between what the kernel shows what the API shows. I maybe wasn't completely clear. We only (currently) do the mapping for parameters. Return values contain the kernel device names. So we don't rewrite the output of guestfs_list_devices, we just suppress the device corresponding to the appliance root filesystem. This both is and isn't a problem at the moment. If you treat the returned strings as opaque blobs, you can pass them back to API functions. However if you want to print them it's a different matter, and lots of tools try to canonicalize the names before printing them, eg: consider the following code in an old version of virt-filesystems: https://github.com/libguestfs/libguestfs/blob/16fdcb4baabcd96dcc16acf1ec7eae34f7304cce/cat/virt-filesystems.c#L707 In fact this canonicalization code was used so widely, I added it to the API in later versions: https://github.com/libguestfs/libguestfs/blob/99759da25c29a7376490074da3d2c22602a2acec/cat/virt-filesystems.c#L642 https://github.com/libguestfs/libguestfs/blob/faf548a4bdd36b476ee6838bfc7d0c134435a331/src/guestfs.c#L1084 For parameters, the mapping process is documented here: http://libguestfs.org/guestfs.3.html#algorithm-for-block-device-name-translation The main reason for mapping parameters is so you can write guestfish scripts which are portable across many versions of libguestfs, eg: guestfish -a foo.img -m /dev/sda1 will just work. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/ -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [libguestfs] Options for hotplugging
On 20/08/12 22:07, Richard W.M. Jones wrote: (1) The raw libvirt option In this one we'd simply provide thin wrappers around virDomainAttachDevice and virDomainDetactDevice, and leave it up to the user to know what they're doing. The problem with this is the hidden appliance disk. We certainly don't want the user to accidentally detach that(!) It's also undesirable for there to be a hole in the naming scheme so that you'd have: /dev/sda - your normal drives /dev/sdb - [/dev/sdc # sorry, you can't use this, we won't tell you why] /dev/sdd - your first hotplugged device As far as I know, the kernel assigns /dev/sdX names on a first-free basis, so there's no way to permanently put the appliance at /dev/sdzzz (if there is, please let me know!) There are numerous reasons not to like this. If there were any reason to want to expose the underlying libvirt api directly I'd suggest it only be a debug option. (2) The slots option In this option you'd have to use null devices to reserve the maximum number of drive slots that you're going to use in the libguestfs handle before launch. Then after launching you'd be allowed to hotplug only those slots. So for example: guestfs_add_drive (g, /dev/null); # reserves /dev/sda guestfs_add_drive (g, /dev/null); # reserves /dev/sdb guestfs_add_drive (g, /dev/null); # reserves /dev/sdc guestfs_launch (g); guestfs_hotplug (g, 1, /tmp/foo); # replaces index 1 == /dev/sdb guestfs_hotplug (g, 3, /tmp/foo); # error! Although ugly, in some ways this is quite attractive. It maps easily into guestfish scripts. You have contiguous device naming. You often know how many drives you'll need in advance, and if you don't then you can reserve up to max_disks-1. Echo Dan's general dislike of this. (3) The serial numbers option This was Dan's suggestion. Hotplugged drives are known only by their serial number. ie. We hotplug them via libvirt using the serial/ field, and then they are accessed using /dev/disk/by-id/serial. This is tempting, but unfortunately it doesn't quite work in stock udev, because the actual name used is: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_SERIAL We could add a custom udev rule to get the path we wanted. Presumably you would specify then serial number when adding the drive? I'm not opposed to this, but it's not the simplest. (4) The rewriting device names option Since we already have the infrastructure to rewrite device names, we could do some complicated and hairy device name rewriting to make names appear continguous, even though there's an hidden appliance drive. This is my least favourite option, mainly because of the complexity, and complexity is bound to lead to bugs. I don't think this rewriting is required. Having a hole in the drive letters isn't a big deal. In fact, I suspect it would be simpler in most code to use returned rather than calculated device names. I'd suggest a very simple api: char * guestfs_hotplug_drive(g, path, opts) This does the same as add_drive, except that it works after launch and returns the api name of the newly added drive. list_devices will return a list with a hole in it. If it isn't already there, we can add some generic code to methods taking a Device parameter to guard against passing in the root device. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [libguestfs] Options for hotplugging
On Tue, Aug 21, 2012 at 11:49:04AM +0100, Matthew Booth wrote: (3) The serial numbers option This was Dan's suggestion. Hotplugged drives are known only by their serial number. ie. We hotplug them via libvirt using the serial/ field, and then they are accessed using /dev/disk/by-id/serial. This is tempting, but unfortunately it doesn't quite work in stock udev, because the actual name used is: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_SERIAL We could add a custom udev rule to get the path we wanted. Presumably you would specify then serial number when adding the drive? I'm not opposed to this, but it's not the simplest. That's right, possibly reusing the name field for this purpose. (4) The rewriting device names option Since we already have the infrastructure to rewrite device names, we could do some complicated and hairy device name rewriting to make names appear continguous, even though there's an hidden appliance drive. This is my least favourite option, mainly because of the complexity, and complexity is bound to lead to bugs. I don't think this rewriting is required. Having a hole in the drive letters isn't a big deal. In fact, I suspect it would be simpler in most code to use returned rather than calculated device names. I'd suggest a very simple api: char * guestfs_hotplug_drive(g, path, opts) This does the same as add_drive, except that it works after launch and returns the api name of the newly added drive. list_devices will return a list with a hole in it. If it isn't already there, we can add some generic code to methods taking a Device parameter to guard against passing in the root device. I looked at this and thought it might be tricky to implement. ie. The implementation would be: - (library) add the drive via libvirt with a generated serial - (appliance) query the appliance to find out the new device name If so, then it would essentially reduce to the serial option, but hiding the serial number. Any simpler way to implement this? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[libvirt] [libguestfs] Options for hotplugging
libguestfs recently added support for virtio-scsi and libvirt, and when these are both available this lets us relatively easily add hotplugging of drives. This email is about how we would present that through the libguestfs API. (a) The current API Currently you have to add drive(s) via guestfs_add_drive* and then call guestfs_launch, ie. your program must look something like this: guestfs_h *g = guestfs_create (); guestfs_add_drive (g, /tmp/disk1.img); guestfs_add_drive (g, /tmp/disk2.img); guestfs_launch (g); After guestfs_launch [the qemu backend is running] you are not allowed to add more drives. The API specifies that you refer to drives in other commands in one of two ways. Either you're allowed to use names like /dev/sda, /dev/sdb etc to refer to the first, second etc drive you added, in the same order that you added them. Or you can call guestfs_list_devices which returns a list of device names, opaque strings that you pass to other functions. In the first case (using /dev/sdX names), some magic already happens translating these to the real names underneath, but currently that magic is just /dev/sdX - /dev/vdX for the virtio case. Note that we cannot change this API or break existing programs. (b) The hidden appliance drive The libguestfs appliance has to have its own root drive. Currently this is added after the user-added drives. For example, if the user adds two drives, then the appliance will appear as /dev/sdc (or /dev/vdc or whatever). Some magic in the bootloader causes the last drive to be used as the root filesystem. This hidden drive doesn't appear in the API -- for example it is suppressed when we generate the result of guestfs_list_devices. (c) /dev/null drives It's always been possible to add /dev/null as a drive via guestfs_add_drive*. This is mainly useful for testing, or if you want to access just those parts of the API which don't require a disk image (for various reasons we force you to add one drive, so if you don't have any drive to add, you can use /dev/null). Current libguestfs treats /dev/null as a magic string and (because of bugs in qemu) substitutes a non-zero sized temporary file. (d) Maximum number of drives With virtio-scsi, this maximum is pretty large -- currently 255 (256 targets less the hidden appliance), but if we used LUNs or even multiple controllers then it'd be almost unlimited. We actually test up to 255, and virt-df will use as many slots as it can. (e) For libguestfs you can assume the latest of everything, qemu, guest kernel, host kernel, libvirt, tools. Any suggestions based on very new features are fine, even proposed features provided there's a working implementation which is likely to go upstream. - - - - Here are some ideas about how we might add hotplugging without breaking existing clients. (1) The raw libvirt option In this one we'd simply provide thin wrappers around virDomainAttachDevice and virDomainDetactDevice, and leave it up to the user to know what they're doing. The problem with this is the hidden appliance disk. We certainly don't want the user to accidentally detach that(!) It's also undesirable for there to be a hole in the naming scheme so that you'd have: /dev/sda - your normal drives /dev/sdb - [/dev/sdc # sorry, you can't use this, we won't tell you why] /dev/sdd - your first hotplugged device As far as I know, the kernel assigns /dev/sdX names on a first-free basis, so there's no way to permanently put the appliance at /dev/sdzzz (if there is, please let me know!) (2) The slots option In this option you'd have to use null devices to reserve the maximum number of drive slots that you're going to use in the libguestfs handle before launch. Then after launching you'd be allowed to hotplug only those slots. So for example: guestfs_add_drive (g, /dev/null); # reserves /dev/sda guestfs_add_drive (g, /dev/null); # reserves /dev/sdb guestfs_add_drive (g, /dev/null); # reserves /dev/sdc guestfs_launch (g); guestfs_hotplug (g, 1, /tmp/foo); # replaces index 1 == /dev/sdb guestfs_hotplug (g, 3, /tmp/foo); # error! Although ugly, in some ways this is quite attractive. It maps easily into guestfish scripts. You have contiguous device naming. You often know how many drives you'll need in advance, and if you don't then you can reserve up to max_disks-1. (3) The serial numbers option This was Dan's suggestion. Hotplugged drives are known only by their serial number. ie. We hotplug them via libvirt using the serial/ field, and then they are accessed using /dev/disk/by-id/serial. This is tempting, but unfortunately it doesn't quite work in stock udev, because the actual name used is: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_SERIAL We could add a custom udev rule to get the path we wanted. (4) The rewriting device names option Since we already have the infrastructure to rewrite device names, we could do some complicated and hairy device name
