I switched libguestfs over to using virtio-scsi. One immediate benefit is support for large numbers of disks: up to 255 because we're using 1 target / disk and we reserve one disk for the appliance, in theory more could be supported if we used LUNs.
This email just contains some notes that may be useful for others trying to use large numbers of disks, and/or virtio-scsi, and/or qemu, and/or Linux guests, all together. * The command line syntax is convoluted. You need to first of all define the virtio-scsi bus once: -device virtio-scsi-pci,id=scsi and then, for each disk, you need these two parameters: -drive file=disk.img,cache=off,format=raw,id=hd0,if=none -device scsi-hd,drive=hd0 (adjust 'id' and 'drive' to be a unique name for each disk) * Linux probes the SCSI bus asynchronously. This was a problem for us because our initramfs doesn't contain udev; it originally insmod'd the virtio-scsi.ko module and expected the disks to be present (note that this works for virtio-blk). Possible solutions include: wait for the disks to become present, or use udev, or do something with devtmpfs, or read out the events from the kernel like udev does. We modified our initramfs to wait for the disk to be seen under /sys/block, that being the easiest change to make at the moment. Also scsi_wait_scan is apparently deprecated. * This describes the naming scheme for > 26 disks under Linux: https://rwmj.wordpress.com/2011/01/09/how-are-linux-drives-named-beyond-drive-26-devsdz/ * Devices are named /dev/sdX instead of /dev/vdX (largely an improvement). * With 256 disks, on fast hardware, with KVM, it takes a Linux guest about 3 seconds to probe the disks, and udev approx another 3 seconds to process its event queue. * With 256 disks, on fast hardware, but using TCG, it takes a Linux guest over 7 minutes to probe the disks. This causes udev to timeout while processing the coldstart events, unless you increase the `udevadm settle --timeout' in the startup scripts. I don't understand why TCG is so much slower, since TCG itself isn't 100 times slower normally. * The overall libguestfs startup time (ie. with two disks) is down fractionally to 3.6 seconds, which is nice. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora