Re: [Qemu-devel] About making QEMU to LIBs!
On Thu, Mar 28, 2019 at 02:45:37PM +0800, Xiao Guangrong wrote: > > > On 3/27/19 5:41 PM, Stefano Garzarella wrote: > > Hi Yang, Xiao, > > Just adding few things, because I'm currently exploring the QEMU modules > > in order to reduce the boot time and the footprint. > > > > Hi Stefan Hajnoczi and Stefano Garzarella, > > The work exploring the QEMU modules looks really good. we will > evaluate it. > > Hmm... how about make these modules usable out of QEMU, it seems > doable, right? But I'm not sure how much effort we need. This requires a stable API, which QEMU currently does not offer. It's the same situation as Linux kernel APIs for kernel modules - function prototypes can be added/removed/changed between kernel releases so third-party code will break. The effort involved in stable APIs is an ongoing maintenance overhead to ensure backwards compatibility, a deprecation policy for removing old interfaces, etc. This makes it harder to change the code and we try to avoid it for that reason. If you contribute your code to qemu.git then stable APIs aren't necessary since there will be build errors if someone changes an API your code is using. Do you think your code fits into the scope of QEMU? Stefan signature.asc Description: PGP signature
Re: [Qemu-devel] About making QEMU to LIBs!
On Thu, Mar 28, 2019 at 02:45:37PM +0800, Xiao Guangrong wrote: > On 3/27/19 5:41 PM, Stefano Garzarella wrote: > > Hi Yang, Xiao, > > Just adding few things, because I'm currently exploring the QEMU modules > > in order to reduce the boot time and the footprint. > > > > Hi Stefan Hajnoczi and Stefano Garzarella, > > The work exploring the QEMU modules looks really good. we will > evaluate it. > > Hmm... how about make these modules usable out of QEMU, it seems > doable, right? But I'm not sure how much effort we need. What use case do you have in mind? These modules use several QEMU internal APIs, so you should provide all of them. > > > > > The idea is to create new QEMU modules (like the block/audio/ui > > drivers) in order to reduce the dependency of QEMU executable to the > > shared libraries that require a lot of time for the initialization, so > > we can postpone their initialization only when we will use them. > > > > Using callgrind I'm profiling the loading of dynamic libraries to > > understand which part of QEMU should be modularized to speed up the boot > > phase when they are not used. > > > > I found that libgnutls, libspice-server and libnuma initialization takes > > most of the time. I disabled these libraries (--disable-numa > > --disable-gnutls > > --disable-spice) to understand the maximum speed up and I gained 10 ms > > during the boot: > > - qemu_init_end: 50.92 ms > > - qemu_init_end: 40.17 ms (without numa, gnutls and spice) > > > > I would start to modularize spice, the library is used in the spice > > drivers (spice-display, spice-chardev, spice-input, etc.) but also in > > the QXL emulation. > > > > Maybe we can synchronize our work :) > > Great work, Stefano! Thanks :) > > Does the benefit come from the fact that your work avoids load some > libraries during exec(), right? So it depends on the size of the > shared library? > Partially. The benefit also depends of the time needed in the init functions of those libraries (__attribute__((constructor))) that in some cases requires a lot of time, but I would consider also the size of libraries. Thanks, Stefano
Re: [Qemu-devel] About making QEMU to LIBs!
On 3/27/19 5:41 PM, Stefano Garzarella wrote: Hi Yang, Xiao, Just adding few things, because I'm currently exploring the QEMU modules in order to reduce the boot time and the footprint. Hi Stefan Hajnoczi and Stefano Garzarella, The work exploring the QEMU modules looks really good. we will evaluate it. Hmm... how about make these modules usable out of QEMU, it seems doable, right? But I'm not sure how much effort we need. The idea is to create new QEMU modules (like the block/audio/ui drivers) in order to reduce the dependency of QEMU executable to the shared libraries that require a lot of time for the initialization, so we can postpone their initialization only when we will use them. Using callgrind I'm profiling the loading of dynamic libraries to understand which part of QEMU should be modularized to speed up the boot phase when they are not used. I found that libgnutls, libspice-server and libnuma initialization takes most of the time. I disabled these libraries (--disable-numa --disable-gnutls --disable-spice) to understand the maximum speed up and I gained 10 ms during the boot: - qemu_init_end: 50.92 ms - qemu_init_end: 40.17 ms (without numa, gnutls and spice) I would start to modularize spice, the library is used in the spice drivers (spice-display, spice-chardev, spice-input, etc.) but also in the QXL emulation. Maybe we can synchronize our work :) Great work, Stefano! Does the benefit come from the fact that your work avoids load some libraries during exec(), right? So it depends on the size of the shared library?
Re: [Qemu-devel] About making QEMU to LIBs!
On Wed, Mar 27, 2019 at 01:33:06PM +0100, Paolo Bonzini wrote: > On 27/03/19 10:41, Stefano Garzarella wrote: > > I found that libgnutls, libspice-server and libnuma initialization takes > > most of the time. I disabled these libraries (--disable-numa > > --disable-gnutls > > --disable-spice) to understand the maximum speed up and I gained 10 ms > > during the boot: > > - qemu_init_end: 50.92 ms > > - qemu_init_end: 40.17 ms (without numa, gnutls and spice) > > libnuma could probably be removed completely, in favor of invoking the > syscalls manually. Yes, actually we only use mbind() in hostmem.c (if I checked well) I'll propose a patch! > > A while ago I talked to the SPICE guys, asking how hard it would be to > separate the server from the relatively small part that is useful in QXL > emulation (removing everything related to network transparency), but the > answer was "very". :( :( > > However, we could conceivably switch to lazily dlopening the SPICE libs. Thanks for your suggestions! Stefano
Re: [Qemu-devel] About making QEMU to LIBs!
On Tue, Mar 26, 2019 at 02:31:00PM +0100, Paolo Bonzini wrote: > On 26/03/19 12:38, Yang Zhong wrote: > > On Tue, Mar 26, 2019 at 10:07:35AM +0100, Paolo Bonzini wrote: > >> Would separate QEMU binaries be a solution? I think I am not as opposed > >> to a "q35-lite" machine type these days, I find it preferrable to share > >> the northbridge and southbridge with Q35 and just get rid of IDE, VGA, > >> IOAPIC, legacy ISA devices etc. The chipset would stay the same as q35 > >> so that we keep secure boot, share the code for ACPI stuff (hotplug), > >> and if the OS needs it we can also add back the RTC. > >> > > Paolo, i am doing NEMU rebase work and will make up these patches for > > upstream. You do not want to add extra machine type for x86 ? There > > are two type of light weight solutions in our intel > > 1) qemu-lite > > PVH has been merged into Qemu 4.0, which seems there is no chance > > for our skip bios solution for upstream? > > 2) NEMU > > Our previous plan is to upstream NEMU's virt platform into Qemu. > > Personally I don't see any advantage in the virt platform compared to > q35-lite. The way you describe q35-lite makes it fairly close to NEMU's virt. Legacy vs hw-reduced ACPI and chipset emulation would be the main differences iiuc. Besides secure boot support (I still need to understand why the latter needs the former, sorry), are there any compelling reasons for keeping ICH9 emulation? Cheers, Samuel.
Re: [Qemu-devel] About making QEMU to LIBs!
On 27/03/19 14:19, Samuel Ortiz wrote: > Hi Paolo, > > On Tue, Mar 26, 2019 at 10:07:35AM +0100, Paolo Bonzini wrote: >> Would separate QEMU binaries be a solution? I think I am not as opposed >> to a "q35-lite" machine type these days, I find it preferrable to share >> the northbridge and southbridge with Q35 and just get rid of IDE, VGA, >> IOAPIC, legacy ISA devices etc. The chipset would stay the same as q35 >> so that we keep secure boot, > Excuse my ignorance, but could you explain why the chipset emulation is > needed for secure boot? Because currently Secure Boot requires SMM (if you don't have SMM, it is not really secure because the key database can be overwritten by untrusted code). Of course you could add SMM support to your virt PCI host bridge, and support for your virt PCI host bridge to OVMF. However, at some point you get into diminishing returns. The situation is more or less the same as for hotplug. In that case, some of the things that ICH9 is doing are still needed even if you get rid of IOAPIC/INTX, so you still need something to will do those things. You can pick either the existing ICH9 or a new ACPI hotplug controller, but the code still has to be there somewhere. Paolo
Re: [Qemu-devel] About making QEMU to LIBs!
Hi Paolo, On Tue, Mar 26, 2019 at 10:07:35AM +0100, Paolo Bonzini wrote: > Would separate QEMU binaries be a solution? I think I am not as opposed > to a "q35-lite" machine type these days, I find it preferrable to share > the northbridge and southbridge with Q35 and just get rid of IDE, VGA, > IOAPIC, legacy ISA devices etc. The chipset would stay the same as q35 > so that we keep secure boot, Excuse my ignorance, but could you explain why the chipset emulation is needed for secure boot? Cheers, Samuel.
Re: [Qemu-devel] About making QEMU to LIBs!
On 27/03/19 10:41, Stefano Garzarella wrote: > I found that libgnutls, libspice-server and libnuma initialization takes > most of the time. I disabled these libraries (--disable-numa --disable-gnutls > --disable-spice) to understand the maximum speed up and I gained 10 ms > during the boot: > - qemu_init_end: 50.92 ms > - qemu_init_end: 40.17 ms (without numa, gnutls and spice) libnuma could probably be removed completely, in favor of invoking the syscalls manually. A while ago I talked to the SPICE guys, asking how hard it would be to separate the server from the relatively small part that is useful in QXL emulation (removing everything related to network transparency), but the answer was "very". :( However, we could conceivably switch to lazily dlopening the SPICE libs. Paolo > I would start to modularize spice, the library is used in the spice > drivers (spice-display, spice-chardev, spice-input, etc.) but also in > the QXL emulation.
Re: [Qemu-devel] About making QEMU to LIBs!
Hi Yang, Xiao, Just adding few things, because I'm currently exploring the QEMU modules in order to reduce the boot time and the footprint. On Tue, Mar 26, 2019 at 09:27:09AM +, Stefan Hajnoczi wrote: > On Tue, Mar 26, 2019 at 03:00:05PM +0800, Xiao Guangrong wrote: > > On 3/26/19 7:18 AM, Paolo Bonzini wrote: > > > On 25/03/19 12:46, Yang Zhong wrote: > > > > Hello all, > > > > > > > > Rust-VMM has started to make all features and common modules to crates, > > > > and CSP can > > > > deploy their VMM on demand. This afternoon, Xiao guangrong and i talked > > > > about the light > > > > weight VM solitions，and we thought QEMU can do same thing like > > > > Rust-vmm, although we can > > > > make all modules or features in QEMU configurable, but making features > > > > and modules to libs > > > > are still valuable for QEMU. Any comments are welcome! thanks! > > > > > > What features/modules were you thinking of? Were you thinking of > > > turning them into dynamically loaded libraries, or spinning them out > > > into separate projects (such as libslirp)? > > > > We are considering make QEMU's device emulations to dynamically libraries > > that include > > virtio devices, IO controller and even vCPU emulations plus some hooks into > > it, and so > > on. > > > > > > > > Also, what is the use case? Is it to reduce the attack surface without > > > having multiple QEMU binaries? > > > > > > > Security is one story we concern， only the essential and audited > > modules/libraries > > can be loaded into memory. > > QEMU's build system supports dynamically loaded "modules" (shared > libraries). You can build all features but only load a subset of them > at runtime. You can also install just a subset of modules on a host. > > Examples are the block drivers and audio drivers. See ./configure and > block/Makefile.objs. QEMU's build system has 'y', 'n', and 'm' (for module). > > So far no emulated devices are built as modules, but this would be a > good area to explore. > The idea is to create new QEMU modules (like the block/audio/ui drivers) in order to reduce the dependency of QEMU executable to the shared libraries that require a lot of time for the initialization, so we can postpone their initialization only when we will use them. Using callgrind I'm profiling the loading of dynamic libraries to understand which part of QEMU should be modularized to speed up the boot phase when they are not used. I found that libgnutls, libspice-server and libnuma initialization takes most of the time. I disabled these libraries (--disable-numa --disable-gnutls --disable-spice) to understand the maximum speed up and I gained 10 ms during the boot: - qemu_init_end: 50.92 ms - qemu_init_end: 40.17 ms (without numa, gnutls and spice) I would start to modularize spice, the library is used in the spice drivers (spice-display, spice-chardev, spice-input, etc.) but also in the QXL emulation. Maybe we can synchronize our work :) Cheers, Stefano
Re: [Qemu-devel] About making QEMU to LIBs!
On 26/03/19 12:38, Yang Zhong wrote: > On Tue, Mar 26, 2019 at 10:07:35AM +0100, Paolo Bonzini wrote: >> Would separate QEMU binaries be a solution? I think I am not as opposed >> to a "q35-lite" machine type these days, I find it preferrable to share >> the northbridge and southbridge with Q35 and just get rid of IDE, VGA, >> IOAPIC, legacy ISA devices etc. The chipset would stay the same as q35 >> so that we keep secure boot, share the code for ACPI stuff (hotplug), >> and if the OS needs it we can also add back the RTC. >> > Paolo, i am doing NEMU rebase work and will make up these patches for > upstream. You do not want to add extra machine type for x86 ? There > are two type of light weight solutions in our intel > 1) qemu-lite > PVH has been merged into Qemu 4.0, which seems there is no chance > for our skip bios solution for upstream? > 2) NEMU > Our previous plan is to upstream NEMU's virt platform into Qemu. Personally I don't see any advantage in the virt platform compared to q35-lite. Other maintainers may disagree, in which case I will happily defer to their judgment. Paolo
Re: [Qemu-devel] About making QEMU to LIBs!
On Tue, Mar 26, 2019 at 10:07:35AM +0100, Paolo Bonzini wrote: > On 26/03/19 08:00, Xiao Guangrong wrote: > > On 3/26/19 7:18 AM, Paolo Bonzini wrote: > >> Also, what is the use case? Is it to reduce the attack surface without > >> having multiple QEMU binaries? > > > > Security is one story we concern， only the essential and audited > > modules/libraries can be loaded into memory. > > > > Another story is that lightweight virt. becomes more and more important > > to run VM-based workload in the public Cloud. However, QEMU is too huge > > and cumbersome to fill our requirements, e.g, QEMU-lites has been being > > developed for a long time but still lacks a way into mainline or > > Intel's nEMU more radically. > > What is your definition of lightweight? To some extent, moving devices > to separate dynamic libraries _adds_ weight for the mechanisms to load > those libraries dynamically. > > Would separate QEMU binaries be a solution? I think I am not as opposed > to a "q35-lite" machine type these days, I find it preferrable to share > the northbridge and southbridge with Q35 and just get rid of IDE, VGA, > IOAPIC, legacy ISA devices etc. The chipset would stay the same as q35 > so that we keep secure boot, share the code for ACPI stuff (hotplug), > and if the OS needs it we can also add back the RTC. > Paolo, i am doing NEMU rebase work and will make up these patches for upstream. You do not want to add extra machine type for x86 ? There are two type of light weight solutions in our intel 1) qemu-lite PVH has been merged into Qemu 4.0, which seems there is no chance for our skip bios solution for upstream? 2) NEMU Our previous plan is to upstream NEMU's virt platform into Qemu. Thanks! Yang > Paolo
Re: [Qemu-devel] About making QEMU to LIBs!
On 3/26/19 5:07 PM, Paolo Bonzini wrote: On 26/03/19 08:00, Xiao Guangrong wrote: On 3/26/19 7:18 AM, Paolo Bonzini wrote: Also, what is the use case? Is it to reduce the attack surface without having multiple QEMU binaries? Security is one story we concern， only the essential and audited modules/libraries can be loaded into memory. Another story is that lightweight virt. becomes more and more important to run VM-based workload in the public Cloud. However, QEMU is too huge and cumbersome to fill our requirements, e.g, QEMU-lites has been being developed for a long time but still lacks a way into mainline or Intel's nEMU more radically. What is your definition of lightweight? To some extent, moving devices to separate dynamic libraries _adds_ weight for the mechanisms to load those libraries dynamically. Lightweight = less resource usage + fast to reach user's self-defined workload. For the new software, these libraries can be statically linked into it. Would separate QEMU binaries be a solution? I think I am not as opposed Separate QEMU binaries mean we need multiple different compiling environments and additional package maintenance on the production... to a "q35-lite" machine type these days, I find it preferrable to share the northbridge and southbridge with Q35 and just get rid of IDE, VGA, IOAPIC, legacy ISA devices etc. The chipset would stay the same as q35 so that we keep secure boot, share the code for ACPI stuff (hotplug), and if the OS needs it we can also add back the RTC. Well, that is a big step forward. :) I'd think making most used components, such as virtio devices, are statically compiled that is the minimal requirements for lightweight scenario. Other components are libraries which can be dlopen() at the runtime, that slow little down to boot the traditional VMs indeed but that is acceptable...
Re: [Qemu-devel] About making QEMU to LIBs!
On Tue, Mar 26, 2019 at 03:00:05PM +0800, Xiao Guangrong wrote: > On 3/26/19 7:18 AM, Paolo Bonzini wrote: > > On 25/03/19 12:46, Yang Zhong wrote: > > > Hello all, > > > > > > Rust-VMM has started to make all features and common modules to crates, > > > and CSP can > > > deploy their VMM on demand. This afternoon, Xiao guangrong and i talked > > > about the light > > > weight VM solitions，and we thought QEMU can do same thing like Rust-vmm, > > > although we can > > > make all modules or features in QEMU configurable, but making features > > > and modules to libs > > > are still valuable for QEMU. Any comments are welcome! thanks! > > > > What features/modules were you thinking of? Were you thinking of > > turning them into dynamically loaded libraries, or spinning them out > > into separate projects (such as libslirp)? > > We are considering make QEMU's device emulations to dynamically libraries > that include > virtio devices, IO controller and even vCPU emulations plus some hooks into > it, and so > on. > > > > > Also, what is the use case? Is it to reduce the attack surface without > > having multiple QEMU binaries? > > > > Security is one story we concern， only the essential and audited > modules/libraries > can be loaded into memory. QEMU's build system supports dynamically loaded "modules" (shared libraries). You can build all features but only load a subset of them at runtime. You can also install just a subset of modules on a host. Examples are the block drivers and audio drivers. See ./configure and block/Makefile.objs. QEMU's build system has 'y', 'n', and 'm' (for module). So far no emulated devices are built as modules, but this would be a good area to explore. Do you think modules satisfy your requirements? > Another story is that lightweight virt. becomes more and more important to > run VM-based > workload in the public Cloud. However, QEMU is too huge and cumbersome to > fill our > requirements, e.g, QEMU-lites has been being developed for a long time but > still lacks > a way into mainline or Intel's nEMU more radically. It's worth noting that NEMU functionality is being upstreamed or inspired upstream work. QEMU recently got Kconfig (better compile-time dependency management) and PVH fast guest kernel loading. If you are interested in lightweight VMs, check out those features. Stefan signature.asc Description: PGP signature
Re: [Qemu-devel] About making QEMU to LIBs!
On 26/03/19 08:00, Xiao Guangrong wrote: > On 3/26/19 7:18 AM, Paolo Bonzini wrote: >> Also, what is the use case? Is it to reduce the attack surface without >> having multiple QEMU binaries? > > Security is one story we concern， only the essential and audited > modules/libraries can be loaded into memory. > > Another story is that lightweight virt. becomes more and more important > to run VM-based workload in the public Cloud. However, QEMU is too huge > and cumbersome to fill our requirements, e.g, QEMU-lites has been being > developed for a long time but still lacks a way into mainline or > Intel's nEMU more radically. What is your definition of lightweight? To some extent, moving devices to separate dynamic libraries _adds_ weight for the mechanisms to load those libraries dynamically. Would separate QEMU binaries be a solution? I think I am not as opposed to a "q35-lite" machine type these days, I find it preferrable to share the northbridge and southbridge with Q35 and just get rid of IDE, VGA, IOAPIC, legacy ISA devices etc. The chipset would stay the same as q35 so that we keep secure boot, share the code for ACPI stuff (hotplug), and if the OS needs it we can also add back the RTC. Paolo
Re: [Qemu-devel] About making QEMU to LIBs!
On 3/26/19 7:18 AM, Paolo Bonzini wrote: On 25/03/19 12:46, Yang Zhong wrote: Hello all, Rust-VMM has started to make all features and common modules to crates, and CSP can deploy their VMM on demand. This afternoon, Xiao guangrong and i talked about the light weight VM solitions，and we thought QEMU can do same thing like Rust-vmm, although we can make all modules or features in QEMU configurable, but making features and modules to libs are still valuable for QEMU. Any comments are welcome! thanks! What features/modules were you thinking of? Were you thinking of turning them into dynamically loaded libraries, or spinning them out into separate projects (such as libslirp)? We are considering make QEMU's device emulations to dynamically libraries that include virtio devices, IO controller and even vCPU emulations plus some hooks into it, and so on. Also, what is the use case? Is it to reduce the attack surface without having multiple QEMU binaries? Security is one story we concern， only the essential and audited modules/libraries can be loaded into memory. Another story is that lightweight virt. becomes more and more important to run VM-based workload in the public Cloud. However, QEMU is too huge and cumbersome to fill our requirements, e.g, QEMU-lites has been being developed for a long time but still lacks a way into mainline or Intel's nEMU more radically. That's why we are going to redesign the userspace VMM, making QEMU to libraries is definitely good to us. Having dynamically linked libraries would go even beyond what rust-vmm does; Rust binaries in the end statically link every crate that they use, rust-vmm's purpose is mostly to share code across VMMs and avoid forks. It may also have a startup time penalty which has to be considered. Rust-VMM is good indeed, but it's not friendly enough for us. QEMU IS A GOOD GUY，no reason to abandon it. :) QEMU is excellent and we are using it in our productions for so many years, its code is trusted as robust. Reusing QEMU's code helps the new software to achieve production-level's quality. Furthermore, we still need QEMU for the traditional workloads, maintaining two totally different code base is not easy for the cloud provider and most developers in the cloud companies are really good at QEMU. So leveraging QEMU and the new software is more convenient for us. Thanks!
Re: [Qemu-devel] About making QEMU to LIBs!
On 25/03/19 12:46, Yang Zhong wrote: > Hello all, > > Rust-VMM has started to make all features and common modules to crates, and > CSP can > deploy their VMM on demand. This afternoon, Xiao guangrong and i talked about > the light > weight VM solitions，and we thought QEMU can do same thing like Rust-vmm, > although we can > make all modules or features in QEMU configurable, but making features and > modules to libs > are still valuable for QEMU. Any comments are welcome! thanks! What features/modules were you thinking of? Were you thinking of turning them into dynamically loaded libraries, or spinning them out into separate projects (such as libslirp)? Also, what is the use case? Is it to reduce the attack surface without having multiple QEMU binaries? Having dynamically linked libraries would go even beyond what rust-vmm does; Rust binaries in the end statically link every crate that they use, rust-vmm's purpose is mostly to share code across VMMs and avoid forks. It may also have a startup time penalty which has to be considered. Thanks, Paolo
[Qemu-devel] About making QEMU to LIBs!
Hello all, Rust-VMM has started to make all features and common modules to crates, and CSP can deploy their VMM on demand. This afternoon, Xiao guangrong and i talked about the light weight VM solitions，and we thought QEMU can do same thing like Rust-vmm, although we can make all modules or features in QEMU configurable, but making features and modules to libs are still valuable for QEMU. Any comments are welcome! thanks! Regards, Yang
