Re: libvirt improvements for bhyve
Craig Rodrigues wrote: > On Sat, Oct 11, 2014 at 11:05 AM, Roman Bogorodskiy > wrote: > > > > > Frankly speaking, it's a hard topic for me and currently I don't have a > > complete idea how to design that. The thing I'm worried about is that as > > far as I understand that exit codes are sort of temporary solution and > > will be dropped in the future. Also, it seems the behaviour is changing > > already, e.g. IIRC on -CURRENT the bhyve proccess doesn't exit on > > reboots (unlike in 10-R). My goal is to support 10-R and I'm not sure > > how to probe if exit codes are available or not (and how to probe if > > e.g. if new exit code will be added in some future version). > > > > Any reason why you are stuck on 10-R? > Unlike other parts of FreeBSD, bhyve is not mature, and is evolving over > time. > Serious users of bhyve are following CURRENT or 10-STABLE. > 10-STABLE has the same error status codes as CURRENT, and 10.1-R is > coming out in a few weeks. > libvirt for bhyve is even less mature than bhyve itself, so tracking > bhyve beyond the release should be OK, but it's up to you. Well, it's not an ultimate goal to support 10-R, but while it's not very painful, I think it's a nice thing to do. Also, it doesn't mean that I'm not adding features available only in -STABLE, I just want user to have a possibility to have a meaningful error messages if certain feature is not support. Also, I forgot to mention one more important issue with the exit code approach. Libvirt doesn't serve as a parent process for the bhyve processes. The reason for that is to allow libvirt restart without having to interrupt the VMs (that's useful for upgrades or e.g. libvirtd crash). I'm not sure how I could track exit codes in such case. IMHO, the good way to solve this problem is to introduce some sort of control socket on the bhyve side that could be used to query VM status and push events about its change. I've been wanting to implement a PoC for that for quite some time, but, unfortunately, haven't got to it yet. > In its current form, my experience has been that bhyve and libvirt are > unusable for serious work, based > on the issues which I've described before for starting/stopping VM's. > bhyve + custom scripting is the only way to go. > > I tried to use libvirt + bhyve for jenkins.freebsd.org, but found it > unusable, even though Jenkins has libvirt > support for starting VM's on demand. I backed away from libvirt, and ended > up writing my > own scripts. I am trying to convince some people to go with libvirt + > bhyve for a simulation/testing system, > but am having a hard time because of all the rough edges, and Linux + KVM + > libvirt looks a lot more > mature and easier to use. > > It's too bad, because I think if you can incorporate a few minor > enhancements to libvirt for bhyve, > it can definitely be a replacement for Linux + KVM + libvirt > > > Neel, Peter, can you comment on the status exit codes from bhyve? > Are those temporary, or will you keep them? There is no other way > for external scripts to detect reset/poweroff. It looks like the confusion > over this issue > is blocking Roman from moving forward with libvirt. > > Thanks. > -- > Craig Roman Bogorodskiy pgpF0qdEbZyie.pgp Description: PGP signature
Re: libvirt improvements for bhyve
On Sat, Oct 11, 2014 at 11:05 AM, Roman Bogorodskiy wrote: > > Frankly speaking, it's a hard topic for me and currently I don't have a > complete idea how to design that. The thing I'm worried about is that as > far as I understand that exit codes are sort of temporary solution and > will be dropped in the future. Also, it seems the behaviour is changing > already, e.g. IIRC on -CURRENT the bhyve proccess doesn't exit on > reboots (unlike in 10-R). My goal is to support 10-R and I'm not sure > how to probe if exit codes are available or not (and how to probe if > e.g. if new exit code will be added in some future version). > Any reason why you are stuck on 10-R? Unlike other parts of FreeBSD, bhyve is not mature, and is evolving over time. Serious users of bhyve are following CURRENT or 10-STABLE. 10-STABLE has the same error status codes as CURRENT, and 10.1-R is coming out in a few weeks. libvirt for bhyve is even less mature than bhyve itself, so tracking bhyve beyond the release should be OK, but it's up to you. In its current form, my experience has been that bhyve and libvirt are unusable for serious work, based on the issues which I've described before for starting/stopping VM's. bhyve + custom scripting is the only way to go. I tried to use libvirt + bhyve for jenkins.freebsd.org, but found it unusable, even though Jenkins has libvirt support for starting VM's on demand. I backed away from libvirt, and ended up writing my own scripts. I am trying to convince some people to go with libvirt + bhyve for a simulation/testing system, but am having a hard time because of all the rough edges, and Linux + KVM + libvirt looks a lot more mature and easier to use. It's too bad, because I think if you can incorporate a few minor enhancements to libvirt for bhyve, it can definitely be a replacement for Linux + KVM + libvirt Neel, Peter, can you comment on the status exit codes from bhyve? Are those temporary, or will you keep them? There is no other way for external scripts to detect reset/poweroff. It looks like the confusion over this issue is blocking Roman from moving forward with libvirt. Thanks. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: libvirt improvements for bhyve
Craig Rodrigues wrote: > Roman, > > I am using bhyve a lot these days, but > I find that I need to write scripts on top of > bhyveload / bhyve in order to work with VM's. > > I would rather use libvirt, because there are many 3rd party > programs to work with VM's that are written for libvirt (especially for > KVM). > > How hard would it be to implement the following: > > (1) The last time I tried libvirt 1.2.7, it did not seem to properly keep > track >if a VM was powered on or shutdown. If I started a VM with libvirt, > then did "shutdown -r now", I couldn't seem to restart the VM, unless > I destroyed the VM in libvirt, and restarted. > This is very inconvenient. > > If you see the latest version of vmrun.sh ( > https://svnweb.freebsd.org/base/head/share/examples/bhyve/vmrun.sh?view=markup > ) > you will see that I added some comments to tell when the VM has > been reset, powered off, etc. by looking at the exit status of bhyve. > > Can we add this logic to libvirt, so that it can properly tell if > a VM is up or not? Frankly speaking, it's a hard topic for me and currently I don't have a complete idea how to design that. The thing I'm worried about is that as far as I understand that exit codes are sort of temporary solution and will be dropped in the future. Also, it seems the behaviour is changing already, e.g. IIRC on -CURRENT the bhyve proccess doesn't exit on reboots (unlike in 10-R). My goal is to support 10-R and I'm not sure how to probe if exit codes are available or not (and how to probe if e.g. if new exit code will be added in some future version). On the other hand, I think I need to check cases when bhyve process vanishes away and probably set VM status to something like 'shutdown / error' or something like that. Also, I think I need to support 'bhyvectl --force-reset' if that's available. I guess that should be safe for all versions of bhyve and I'll add it to my todo list. > (2) "bhyveload -e" allows specifying environment variables to loader. > Can we have a way to specify loader environment variables, > in the libvirt XML config? > > (3) "grub-bhyve" is needed to boot VM's for Linux. > Can we specify an option to choose between "bhyveload" and > "grub-bhyve"? For 2-3 I think bootloader should be honored from the XML file: http://libvirt.org/formatdomain.html#elementsOSBootloader I'll add it to my todo list as well. > (4) bhyve has many options for specifying which PCI slots to use, > and also PCI passthru ( https://wiki.freebsd.org/bhyve/pci_passthru ) > Can this be specified in libvirt? Currently you can specify PCI slot and function by using 'address' tag in the device element (e.g. disk or interface), for example: PCI passthrough is not supported as I've never looked at that, will need to take a look. > > For me, without (1), libvirt is mostly unusable for bhyve. (2) - (4) would > be > nice to have. However, if we could fix libvirt with these things, > then a lot of things I do now in scripts, I could do in libvirt. > I would much rather use libvirt, because then I could take advantage of > a lot of libvirt software targeted towards KVM. > > Thanks. > -- > Craig Roman Bogorodskiy pgpSxXSIXNmRB.pgp Description: PGP signature
libvirt improvements for bhyve
Roman, I am using bhyve a lot these days, but I find that I need to write scripts on top of bhyveload / bhyve in order to work with VM's. I would rather use libvirt, because there are many 3rd party programs to work with VM's that are written for libvirt (especially for KVM). How hard would it be to implement the following: (1) The last time I tried libvirt 1.2.7, it did not seem to properly keep track if a VM was powered on or shutdown. If I started a VM with libvirt, then did "shutdown -r now", I couldn't seem to restart the VM, unless I destroyed the VM in libvirt, and restarted. This is very inconvenient. If you see the latest version of vmrun.sh ( https://svnweb.freebsd.org/base/head/share/examples/bhyve/vmrun.sh?view=markup ) you will see that I added some comments to tell when the VM has been reset, powered off, etc. by looking at the exit status of bhyve. Can we add this logic to libvirt, so that it can properly tell if a VM is up or not? (2) "bhyveload -e" allows specifying environment variables to loader. Can we have a way to specify loader environment variables, in the libvirt XML config? (3) "grub-bhyve" is needed to boot VM's for Linux. Can we specify an option to choose between "bhyveload" and "grub-bhyve"? (4) bhyve has many options for specifying which PCI slots to use, and also PCI passthru ( https://wiki.freebsd.org/bhyve/pci_passthru ) Can this be specified in libvirt? For me, without (1), libvirt is mostly unusable for bhyve. (2) - (4) would be nice to have. However, if we could fix libvirt with these things, then a lot of things I do now in scripts, I could do in libvirt. I would much rather use libvirt, because then I could take advantage of a lot of libvirt software targeted towards KVM. Thanks. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
