Re: [libvirt] (no subject)
On Fri, 2011-12-09 at 20:30 +0800, Osier Yang wrote: > By the way, nobody is interested in kvmtool privodes a way to for > external apps to get the capabilities? Do we still want to suffer > from parsing the capabilities ourselves in future just like what > we do for qemu? :-) I think the feature makes sense especially if it simplifies libvirt. Pekka -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [libvirt] (no subject)
On 2011年12月07日 17:16, Daniel P. Berrange wrote: On Wed, Dec 07, 2011 at 08:21:06AM +0200, Sasha Levin wrote: On Tue, 2011-12-06 at 14:38 +, Daniel P. Berrange wrote: On Fri, Nov 11, 2011 at 07:56:58PM +0800, Osier Yang wrote: * KVM tool manages the network completely itself (with DHCP support?), no way to configure, except specify the modes (user|tap|none). I have not test it yet, but it should need explicit script to setup the network rules(e.g. NAT) for the guest access outside world. Anyway, there is no way for libvirt to control the guest network. If KVM tool support TAP devices, can't be do whatever we like with that just by passing in a configured TAP device from libvir ? KVM tool currently creates and configures the TAP devices it uses, it shouldn't be an issue to have it use a TAP fd passed to it either. How does libvirt do it? Create a /dev/tapX on it's own and pass the fd to the hypervisor? Yes, libvirt opens a /dev/tap device (or a macvtap device for VEPA mode), adds it to the neccessary bridge, and/or configures VEPA, etc and then passes the FD to the hypervisor, with a ARGV parameter to tell the HV which FD is being passed. * console connection is implemented by setup ptys in libvirt, stdout/stderr of kvm tool process is redirected to the master pty, and libvirt connects to the slave pty. This works fine now, but it might be better if kvm tool could provide more advanced console mechanisms. Just like QEMU does? This sounds good enough for now. KVM tools does a redirection to a PTY, which at that point could be redirected to anywhere the user wants. What features might be interesting to do on top of that? I presume that Osier is just comparing with the features QEMU has available for chardevs config, which include - PTYs - UNIX sockets - TCP sockets - UDP sockets - FIFO pipe - Plain file (output only obviously, but useful for logging) Yes, that's what I meant. :-) libvirt doesn't specifically need any of them, but it can support those options if they exist. Yes, these won't prevent us, I just meant it will be great if they are supported. * Not much ways existed yet for external apps or user to query the guest informations. But this might be changed soon per KVM tool grows up quickly. What sort of guest info are you thinking about ? The most immediate pieces of info I can imagine we need are - Mapping between PIDs and vCPU threads - Current balloon driver value Those are pretty easily added using the IPC interface I've mentioned above. For example, 'kvm balloon' and 'kvm stat' will return a lot of info out of the balloon driver (including the memory stats VQ - which afaik we're probably the only ones who actually do that (but I might be wrong) :) Ok, that sounds sufficient for the balloon info. Regards, Daniel -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [libvirt] (no subject)
On 2011年12月07日 14:21, Sasha Levin wrote: On Tue, 2011-12-06 at 14:38 +, Daniel P. Berrange wrote: On Fri, Nov 11, 2011 at 07:56:58PM +0800, Osier Yang wrote: * Lacks of options for user's configuration, such as "-vnc", there is no option for user to configure the properties for the "vnc", such as the port. It hides things, doesn't provide ways to query the properties too, this causes problems for libvirt to add the vnc support, as vnc clients such as virt-manager, virt-viewer, have no way to connect the guest. Even vncviewer can't. Being able to specify a VNC port of libvirt's choosing is pretty much mandatory to be able to support that.In addition being able to specify the bind address is important to be able to control security. eg to only bind to 127.0.0.1, or only to certain NICs in a multi-NIC host. I'll add that feature in the next couple of days. * KVM tool manages the network completely itself (with DHCP support?), no way to configure, except specify the modes (user|tap|none). I have not test it yet, but it should need explicit script to setup the network rules(e.g. NAT) for the guest access outside world. Anyway, there is no way for libvirt to control the guest network. If KVM tool support TAP devices, can't be do whatever we like with that just by passing in a configured TAP device from libvir ? KVM tool currently creates and configures the TAP devices it uses, it shouldn't be an issue to have it use a TAP fd passed to it either. How does libvirt do it? Create a /dev/tapX on it's own and pass the fd to the hypervisor? * There is a gap about the domain status between KVM tool and libvirt, it's caused by KVM tool unlink()s the guest socket when user exits from console (both text and graphic), but libvirt still think the guest is running. Being able to reliably detect shutdown/exit of the KVM too is a very important tasks, and we can't rely on waitpid/SIG_CHLD because we want to daemonize all instances wrt libvirtd. In the QEMU driver we keep open a socket to the monitor, and when we see an I/O error / POLLHUP on the socket we know that QEMU has quit. What is this guest socket used for ? Could libvirt keep open a connection to it ? It's being used for communication with the IPC sub-commands (like 'kvm list', 'kvm debug', etc). It's basically the server in a server-client setup used to signal the hypervisor to do things. Theres also no problem with keeping an open connection to it. I'll update the codes to use it. One other option would be to use inotify to watch for deletion of the guest socket in the filesystem. This is sortof what we do with the UML driver. * KVM tool uses $HOME/.kvm_tool as the state dir, and no way to configure, I made a small patch to allow KVM tool accept a ENV variable, which is "KVM_STATE_DIR", it's used across the driver. I made a simple patch against kvm tool to let the whole patches work. See "[PATCH] kvm tools.". As generally we want the state dir of a driver can be "/var/run/libvirt/kvmtool/..." for root user or "$HOME/.libvirt/kvmtool/run" for non-root user. What does it do with the state dir ? Is that just for storing the guest socket ? afaik that patch should be already in as well. It does two things in the state dir: - Store sockets. - KVM tools has a feature which lets a user boot a guest based on virtio-9p which lets him see a system which is an exact copy of the host. This makes testing of programs and sandboxing very easy. The state files required for that are stored in that dir as well. With QEMU we chose $HOME/.libvirt/qemu or /var/run/libvirt because there was no policy set by QEMU itself. If KVM tool has a policy for where it stores its state, we should just use that, and not try to force it into a libvirt specific location. In a privileged libvirtd instace, we should aim to still have kvmtool itself run as an unprivilegd user / group , eg 'kvmtool:kvmtool' And we could set the home dir of that user to /var/lib/kvmtool * kvmtoolGetVersion is just broken now, as what "./kvm version" returns is something like "3.0.rc5.873.gb73216", however, libvirt wants things like "2.6.40.6-0". This might be not a problem as long as KVM tool has a official package. The version numbers libvirt reports for hypervisors are pretty meaningless really. In that example you give I'd just report '3.0' as the version from libvirt. Anything that relies on these versions from libvirt is doomed to be broken anyway. The version is just a 'git describe' of the kernel tree in which it was built, so if you build KVM tools from an "official" kernel tree* you'll also get pretty versions :) * After KVM tools is merged... * console connection is implemented by setup ptys in libvirt, stdout/stderr of kvm tool process is redirected to the master pty, and libvirt connects to the slave pty. This
Re: [libvirt] (no subject)
On 2011年12月06日 22:38, Daniel P. Berrange wrote: On Fri, Nov 11, 2011 at 07:56:58PM +0800, Osier Yang wrote: Hi, all This is a basic implementation of libvirt Native Linux KVM Tool driver. Note that this is just made with my own interest and spare time, it's not an endorsement/effort by Red Hat, and it isn't supported by Red Hat officially. Basically, the driver is designed as *stateful*, as KVM tool doesn't maintain any info about the guest except a socket which for its own IPC. And it's implemented by using KVM tool binary, which is name "kvm" currently, along with cgroup controllers "cpuacct", and "memory" support. And as one of KVM tool's pricinple is to allow both the non-root and root user to play with. The driver is designed to support root and non-root too, just like QEMU does. Example of the connection URI: virsh -c kvmtool:///system virsh -c kvmtool:///session virsh -c kvmtool+unix:///system virsh -c kvmtool+unix:///session The implementation can support more or less than 15 virsh commands currently, including basic domain cycle operations (define/undefine, start/destroy, suspend/resume, console, setmem, schedinfo, dumpxml, ,autostart, dominfo, etc.) About the domain configuration: * "kernel": must be specified as KVM tool only support boots from the kernel currently (no integration with BIOS app yet). * "disk": only virtio bus is supported, and device type must be 'disk'. * "serial/console": only one console is supported, of type serial or virtio (can extend to support multiple console as long as kvm tool supports, libvirt already supported mutiple console, see upstream commit 0873b688c). * "p9fs": only support specifying the source dir, and mount tag, only type of 'mount' is supported. * "memballoon": only virtio is supported, and there is no way to config the addr. * Multiple "disk" and "p9fs" is supported. * Graphics and network are not supported, will explain below. Please see "[PATCH 7/8]" for an example of the domain config. (which contains all the XMLs supported by current implementation). The problems of Native Linux KVM Tool from libvirt p.o.v: * Some destros package "qemu-kvm" as "kvm", also "kvm" is a long established name for "KVM" itself, so naming the project as "kvm" might be not a good idea. I assume it will be named as "kvmtool" in this implementation, never mind this if you don't like that, it can be updated easily. :-) Yeah, naming the binary 'kvm' is just madness. I'd strongly recommend using 'kvmtool' as the binary name to avoid confusion with existing 'kvm' binaries based on QEMU. * It still doesn't have an official package yet, even no "make install". means we have no way to check the dependancy and do the checking when 'configure'. I assume it will be installed as "/usr/bin/kvmtool" in this implementation. This is the main reason which can prevents upstream libvirt accepting the patches I guess. Ok, not really a problem - we do similar for the regular QEMU driver. * Lacks of options for user's configuration, such as "-vnc", there is no option for user to configure the properties for the "vnc", such as the port. It hides things, doesn't provide ways to query the properties too, this causes problems for libvirt to add the vnc support, as vnc clients such as virt-manager, virt-viewer, have no way to connect the guest. Even vncviewer can't. Being able to specify a VNC port of libvirt's choosing is pretty much mandatory to be able to support that.In addition being able to specify the bind address is important to be able to control security. eg to only bind to 127.0.0.1, or only to certain NICs in a multi-NIC host. * KVM tool manages the network completely itself (with DHCP support?), no way to configure, except specify the modes (user|tap|none). I have not test it yet, but it should need explicit script to setup the network rules(e.g. NAT) for the guest access outside world. Anyway, there is no way for libvirt to control the guest network. If KVM tool support TAP devices, can't be do whatever we like with that just by passing in a configured TAP device from libvir ? * There is a gap about the domain status between KVM tool and libvirt, it's caused by KVM tool unlink()s the guest socket when user exits from console (both text and graphic), but libvirt still think the guest is running. Being able to reliably detect shutdown/exit of the KVM too is a very important tasks, and we can't rely on waitpid/SIG_CHLD because we want to daemonize all instances wrt libvirtd. In the QEMU driver we keep open a socket to the monitor, and when we see an I/O error / POLLHUP on the socket we know that QEMU has quit. What is this guest socket used for ? Could libvirt keep open a connection to it ? One other option would be to use inotify to watch for deletion of the guest
Re: [libvirt] (no subject)
On Wed, Dec 07, 2011 at 08:21:06AM +0200, Sasha Levin wrote: > On Tue, 2011-12-06 at 14:38 +, Daniel P. Berrange wrote: > > On Fri, Nov 11, 2011 at 07:56:58PM +0800, Osier Yang wrote: > > > * KVM tool manages the network completely itself (with DHCP support?), > > > no way to configure, except specify the modes (user|tap|none). I > > > have not test it yet, but it should need explicit script to setup > > > the network rules(e.g. NAT) for the guest access outside world. > > > Anyway, there is no way for libvirt to control the guest network. > > > > If KVM tool support TAP devices, can't be do whatever we like with > > that just by passing in a configured TAP device from libvir ? > > KVM tool currently creates and configures the TAP devices it uses, it > shouldn't be an issue to have it use a TAP fd passed to it either. > > How does libvirt do it? Create a /dev/tapX on it's own and pass the fd > to the hypervisor? Yes, libvirt opens a /dev/tap device (or a macvtap device for VEPA mode), adds it to the neccessary bridge, and/or configures VEPA, etc and then passes the FD to the hypervisor, with a ARGV parameter to tell the HV which FD is being passed. > > > * console connection is implemented by setup ptys in libvirt, > > > stdout/stderr > > > of kvm tool process is redirected to the master pty, and libvirt > > > connects > > > to the slave pty. This works fine now, but it might be better if kvm > > > tool could provide more advanced console mechanisms. Just like QEMU > > > does? > > > > This sounds good enough for now. > > KVM tools does a redirection to a PTY, which at that point could be > redirected to anywhere the user wants. > > What features might be interesting to do on top of that? I presume that Osier is just comparing with the features QEMU has available for chardevs config, which include - PTYs - UNIX sockets - TCP sockets - UDP sockets - FIFO pipe - Plain file (output only obviously, but useful for logging) libvirt doesn't specifically need any of them, but it can support those options if they exist. > > > * Not much ways existed yet for external apps or user to query the guest > > > informations. But this might be changed soon per KVM tool grows up > > > quickly. > > > > What sort of guest info are you thinking about ? The most immediate > > pieces of info I can imagine we need are > > > > - Mapping between PIDs and vCPU threads > > - Current balloon driver value > > Those are pretty easily added using the IPC interface I've mentioned > above. For example, 'kvm balloon' and 'kvm stat' will return a lot of > info out of the balloon driver (including the memory stats VQ - which > afaik we're probably the only ones who actually do that (but I might be > wrong) :) Ok, that sounds sufficient for the balloon info. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [libvirt] (no subject)
On Tue, 2011-12-06 at 14:38 +, Daniel P. Berrange wrote: > On Fri, Nov 11, 2011 at 07:56:58PM +0800, Osier Yang wrote: > > * Lacks of options for user's configuration, such as "-vnc", there > > is no option for user to configure the properties for the "vnc", > > such as the port. It hides things, doesn't provide ways to query > > the properties too, this causes problems for libvirt to add the > > vnc support, as vnc clients such as virt-manager, virt-viewer, > > have no way to connect the guest. Even vncviewer can't. > > Being able to specify a VNC port of libvirt's choosing is pretty > much mandatory to be able to support that.In addition being able > to specify the bind address is important to be able to control > security. eg to only bind to 127.0.0.1, or only to certain NICs > in a multi-NIC host. I'll add that feature in the next couple of days. > > > * KVM tool manages the network completely itself (with DHCP support?), > > no way to configure, except specify the modes (user|tap|none). I > > have not test it yet, but it should need explicit script to setup > > the network rules(e.g. NAT) for the guest access outside world. > > Anyway, there is no way for libvirt to control the guest network. > > If KVM tool support TAP devices, can't be do whatever we like with > that just by passing in a configured TAP device from libvir ? KVM tool currently creates and configures the TAP devices it uses, it shouldn't be an issue to have it use a TAP fd passed to it either. How does libvirt do it? Create a /dev/tapX on it's own and pass the fd to the hypervisor? > > > * There is a gap about the domain status between KVM tool and libvirt, > > it's caused by KVM tool unlink()s the guest socket when user exits > > from console (both text and graphic), but libvirt still think the > > guest is running. > > Being able to reliably detect shutdown/exit of the KVM too is > a very important tasks, and we can't rely on waitpid/SIG_CHLD > because we want to daemonize all instances wrt libvirtd. > > In the QEMU driver we keep open a socket to the monitor, and > when we see an I/O error / POLLHUP on the socket we know that > QEMU has quit. > > What is this guest socket used for ? Could libvirt keep open a > connection to it ? It's being used for communication with the IPC sub-commands (like 'kvm list', 'kvm debug', etc). It's basically the server in a server-client setup used to signal the hypervisor to do things. Theres also no problem with keeping an open connection to it. > One other option would be to use inotify to watch for deletion > of the guest socket in the filesystem. This is sortof what we > do with the UML driver. > > > * KVM tool uses $HOME/.kvm_tool as the state dir, and no way to configure, > > I made a small patch to allow KVM tool accept a ENV variable, > > which is "KVM_STATE_DIR", it's used across the driver. I made a > > simple patch against kvm tool to let the whole patches work. See > > "[PATCH] kvm tools.". As generally we want the state dir of > > a driver can be "/var/run/libvirt/kvmtool/..." for root user or > > "$HOME/.libvirt/kvmtool/run" for non-root user. > > What does it do with the state dir ? Is that just for storing the > guest socket ? afaik that patch should be already in as well. It does two things in the state dir: - Store sockets. - KVM tools has a feature which lets a user boot a guest based on virtio-9p which lets him see a system which is an exact copy of the host. This makes testing of programs and sandboxing very easy. The state files required for that are stored in that dir as well. > > With QEMU we chose $HOME/.libvirt/qemu or /var/run/libvirt because > there was no policy set by QEMU itself. If KVM tool has a policy > for where it stores its state, we should just use that, and not > try to force it into a libvirt specific location. > > In a privileged libvirtd instace, we should aim to still have > kvmtool itself run as an unprivilegd user / group , eg 'kvmtool:kvmtool' > And we could set the home dir of that user to /var/lib/kvmtool > > > * kvmtoolGetVersion is just broken now, as what "./kvm version" returns > > is something like "3.0.rc5.873.gb73216", however, libvirt wants things > > like "2.6.40.6-0". This might be not a problem as long as KVM tool > > has a official package. > > The version numbers libvirt reports for hypervisors are pretty > meaningless really. In that example you give I'd just report '3.0' > as the version from libvirt. Anything that relies on these versions > from libvirt is doomed to be broken anyway. The version is just a 'git describe' of the kernel tree in which it was built, so if you build KVM tools from an "official" kernel tree* you'll also get pretty versions :) * After KVM tools is merged... > > > * console connection is implemented by setup ptys in libvirt, > > stdout/stderr > > of kvm tool process is redir
Re: [libvirt] (no subject)
On Fri, Nov 11, 2011 at 07:56:58PM +0800, Osier Yang wrote: > Hi, all > > This is a basic implementation of libvirt Native Linux KVM > Tool driver. Note that this is just made with my own interest > and spare time, it's not an endorsement/effort by Red Hat, > and it isn't supported by Red Hat officially. > > Basically, the driver is designed as *stateful*, as KVM tool > doesn't maintain any info about the guest except a socket which > for its own IPC. And it's implemented by using KVM tool binary, > which is name "kvm" currently, along with cgroup controllers > "cpuacct", and "memory" support. And as one of KVM tool's > pricinple is to allow both the non-root and root user to play with. > The driver is designed to support root and non-root too, just > like QEMU does. Example of the connection URI: > > virsh -c kvmtool:///system > virsh -c kvmtool:///session > virsh -c kvmtool+unix:///system > virsh -c kvmtool+unix:///session > > The implementation can support more or less than 15 virsh commands > currently, including basic domain cycle operations (define/undefine, > start/destroy, suspend/resume, console, setmem, schedinfo, dumpxml, > ,autostart, dominfo, etc.) > > About the domain configuration: > * "kernel": must be specified as KVM tool only support boots > from the kernel currently (no integration with BIOS app yet). > > * "disk": only virtio bus is supported, and device type must be 'disk'. > > * "serial/console": only one console is supported, of type serial or > virtio (can extend to support multiple console as long as kvm tool > supports, libvirt already supported mutiple console, see upstream > commit 0873b688c). > > * "p9fs": only support specifying the source dir, and mount tag, only > type of 'mount' is supported. > > * "memballoon": only virtio is supported, and there is no way > to config the addr. > > * Multiple "disk" and "p9fs" is supported. > > * Graphics and network are not supported, will explain below. > > Please see "[PATCH 7/8]" for an example of the domain config. (which > contains all the XMLs supported by current implementation). > > The problems of Native Linux KVM Tool from libvirt p.o.v: > > * Some destros package "qemu-kvm" as "kvm", also "kvm" is a long > established name for "KVM" itself, so naming the project as > "kvm" might be not a good idea. I assume it will be named > as "kvmtool" in this implementation, never mind this if you > don't like that, it can be updated easily. :-) Yeah, naming the binary 'kvm' is just madness. I'd strongly recommend using 'kvmtool' as the binary name to avoid confusion with existing 'kvm' binaries based on QEMU. > * It still doesn't have an official package yet, even no "make install". > means we have no way to check the dependancy and do the checking > when 'configure'. I assume it will be installed as "/usr/bin/kvmtool" > in this implementation. This is the main reason which can prevents > upstream libvirt accepting the patches I guess. Ok, not really a problem - we do similar for the regular QEMU driver. > * Lacks of options for user's configuration, such as "-vnc", there > is no option for user to configure the properties for the "vnc", > such as the port. It hides things, doesn't provide ways to query > the properties too, this causes problems for libvirt to add the > vnc support, as vnc clients such as virt-manager, virt-viewer, > have no way to connect the guest. Even vncviewer can't. Being able to specify a VNC port of libvirt's choosing is pretty much mandatory to be able to support that.In addition being able to specify the bind address is important to be able to control security. eg to only bind to 127.0.0.1, or only to certain NICs in a multi-NIC host. > * KVM tool manages the network completely itself (with DHCP support?), > no way to configure, except specify the modes (user|tap|none). I > have not test it yet, but it should need explicit script to setup > the network rules(e.g. NAT) for the guest access outside world. > Anyway, there is no way for libvirt to control the guest network. If KVM tool support TAP devices, can't be do whatever we like with that just by passing in a configured TAP device from libvir ? > * There is a gap about the domain status between KVM tool and libvirt, > it's caused by KVM tool unlink()s the guest socket when user exits > from console (both text and graphic), but libvirt still think the > guest is running. Being able to reliably detect shutdown/exit of the KVM too is a very important tasks, and we can't rely on waitpid/SIG_CHLD because we want to daemonize all instances wrt libvirtd. In the QEMU driver we keep open a socket to the monitor, and when we see an I/O error / POLLHUP on the socket we know that QEMU has quit. What is this guest socket used for ? Could libvirt keep open a connection to it ? One other option would
