qemu no-shutdown feature in libvirt
Hi developers, Thanks for developing and maintaining such a great tool. I'm new and I'm trying to use the no-shutdown feature of qemu with libvirt. I've confirmed with the user list that currently there's no such feature in libvirt that supports what I want to do, so I'm posting here and see if anyone is aware of any existing project that can achieve this or anything under development to avoid re-implementing the wheel. Here is what I'm doing on qemu: When I start a VM with the qemu command I can specify the -no-shutdown flag so that my qemu process doesn't quit even if I shutdown the VM from the inside (issue shutdown or halt command inside VM). The VM is shutdown but the qemu process is not killed so that the jobs I submitted before for example backup (drive-backup) can continue to run even after VM gets shutdown by the user, and I'm able to check the status of the job through qmp commands (by communicating with the qemu process). Is anyone aware of any project that is currently implementing this functionality? Thanks for reading. Thanks, Luna
Re: qemu no-shutdown feature in libvirt
On Mon, Nov 30, 2020 at 14:45:18 -0500, Luna Xu wrote: > Hi developers, Hi, [...] > When I start a VM with the qemu command I can specify the -no-shutdown > flag so that my qemu process doesn't quit even if I shutdown the VM > from the inside (issue shutdown or halt command inside VM). The VM is > shutdown but the qemu process is not killed so that the jobs I > submitted before for example backup (drive-backup) can continue to run > even after VM gets shutdown by the user, and I'm able to check the > status of the job through qmp commands (by communicating with the qemu > process). As I've noted in my reply on libvirt-users, we currently don't support this on the shutdown part of the VM lifecycle. On the startup part you can use VIR_DOMAIN_START_PAUSED for a VM which didn't yet excecute any guest instructions. > Is anyone aware of any project that is currently implementing this > functionality? Thanks for reading. I don't think that there's anybody working on it. If you are interested in implementing the feature the idea would be to add a 'pause' action for pause along with a new 'virDomainPausedReason' such as VIR_DOMAIN_PAUSED_POWEROFF and implement the lifecycle transition from PAUSED -> RUNNING via qemuDomainResume, which would in this case have to issue a 'system-reset' qmp command and emit the appropriate events.
Re: qemu no-shutdown feature in libvirt
Hi Peter, Thank you again for replying here. These are great instructions. I will take a look into these. Thank you! Thanks, Luna On Tue, Dec 1, 2020 at 9:34 AM Peter Krempa wrote: > > On Mon, Nov 30, 2020 at 14:45:18 -0500, Luna Xu wrote: > > Hi developers, > > Hi, > > [...] > > > When I start a VM with the qemu command I can specify the -no-shutdown > > flag so that my qemu process doesn't quit even if I shutdown the VM > > from the inside (issue shutdown or halt command inside VM). The VM is > > shutdown but the qemu process is not killed so that the jobs I > > submitted before for example backup (drive-backup) can continue to run > > even after VM gets shutdown by the user, and I'm able to check the > > status of the job through qmp commands (by communicating with the qemu > > process). > > As I've noted in my reply on libvirt-users, we currently don't support > this on the shutdown part of the VM lifecycle. > > On the startup part you can use VIR_DOMAIN_START_PAUSED for a VM which > didn't yet excecute any guest instructions. > > > Is anyone aware of any project that is currently implementing this > > functionality? Thanks for reading. > > I don't think that there's anybody working on it. > > If you are interested in implementing the feature the idea would be to > add a 'pause' action for pause along with a > new 'virDomainPausedReason' such as VIR_DOMAIN_PAUSED_POWEROFF and > implement the lifecycle transition from PAUSED -> RUNNING via > qemuDomainResume, which would in this case have to issue a > 'system-reset' qmp command and emit the appropriate events. >
Re: qemu no-shutdown feature in libvirt
On Tue, Dec 01, 2020 at 15:34:05 +0100, Peter Krempa wrote: > On Mon, Nov 30, 2020 at 14:45:18 -0500, Luna Xu wrote: > > Hi developers, > > Hi, > > [...] > > > When I start a VM with the qemu command I can specify the -no-shutdown > > flag so that my qemu process doesn't quit even if I shutdown the VM > > from the inside (issue shutdown or halt command inside VM). The VM is > > shutdown but the qemu process is not killed so that the jobs I > > submitted before for example backup (drive-backup) can continue to run > > even after VM gets shutdown by the user, and I'm able to check the > > status of the job through qmp commands (by communicating with the qemu > > process). > > As I've noted in my reply on libvirt-users, we currently don't support > this on the shutdown part of the VM lifecycle. > > On the startup part you can use VIR_DOMAIN_START_PAUSED for a VM which > didn't yet excecute any guest instructions. > > > Is anyone aware of any project that is currently implementing this > > functionality? Thanks for reading. > > I don't think that there's anybody working on it. > > If you are interested in implementing the feature the idea would be to > add a 'pause' action for pause along with a > new 'virDomainPausedReason' such as VIR_DOMAIN_PAUSED_POWEROFF and > implement the lifecycle transition from PAUSED -> RUNNING via > qemuDomainResume, which would in this case have to issue a > 'system-reset' qmp command and emit the appropriate events. This is weird, the domain was paused just before being shut down so resuming it should logically just resume the paused process and finish the shutdown. On the other hand, domain shutdown as a result of calling resume is weird too. So how about just adding a check to qemuDomainResume to report an error if you try to resume a domain which was paused for shutdown? Jirka
Re: qemu no-shutdown feature in libvirt
Hi Jiri, Thanks for replying. Actually the no-shutdown flag behavior in qemu is not pausing the VM shutdown, the VM shuts down normally (like you see the linux message saying it's powered off in the tty console, and you can't log in again), but the qemu process stays to finish the async (coroutine) jobs in the background. If the libvirt could take the no-shutdown flag and pass it to qemu, qemu would handle this automatically but somehow libvirt shouldn't kill the qemu process but wait for the process to finish working and exit normally itself. If the VM is started again before the qemu process quits itself, we would need to find a way maybe wait for the last qemu process to quit before launching a new one, or reuse the qmeu process just start the VM virtualization coroutine again inside the same qemu process. Thanks, Luna On Tue, Dec 1, 2020 at 11:09 AM Jiri Denemark wrote: > > On Tue, Dec 01, 2020 at 15:34:05 +0100, Peter Krempa wrote: > > On Mon, Nov 30, 2020 at 14:45:18 -0500, Luna Xu wrote: > > > Hi developers, > > > > Hi, > > > > [...] > > > > > When I start a VM with the qemu command I can specify the -no-shutdown > > > flag so that my qemu process doesn't quit even if I shutdown the VM > > > from the inside (issue shutdown or halt command inside VM). The VM is > > > shutdown but the qemu process is not killed so that the jobs I > > > submitted before for example backup (drive-backup) can continue to run > > > even after VM gets shutdown by the user, and I'm able to check the > > > status of the job through qmp commands (by communicating with the qemu > > > process). > > > > As I've noted in my reply on libvirt-users, we currently don't support > > this on the shutdown part of the VM lifecycle. > > > > On the startup part you can use VIR_DOMAIN_START_PAUSED for a VM which > > didn't yet excecute any guest instructions. > > > > > Is anyone aware of any project that is currently implementing this > > > functionality? Thanks for reading. > > > > I don't think that there's anybody working on it. > > > > If you are interested in implementing the feature the idea would be to > > add a 'pause' action for pause along with a > > new 'virDomainPausedReason' such as VIR_DOMAIN_PAUSED_POWEROFF and > > > implement the lifecycle transition from PAUSED -> RUNNING via > > qemuDomainResume, which would in this case have to issue a > > 'system-reset' qmp command and emit the appropriate events. > > This is weird, the domain was paused just before being shut down so > resuming it should logically just resume the paused process and finish > the shutdown. On the other hand, domain shutdown as a result of calling > resume is weird too. > > So how about just adding a check to qemuDomainResume to report an error > if you try to resume a domain which was paused for shutdown? > > Jirka
Re: qemu no-shutdown feature in libvirt
On Tue, Dec 01, 2020 at 05:09:26PM +0100, Jiri Denemark wrote: > On Tue, Dec 01, 2020 at 15:34:05 +0100, Peter Krempa wrote: > > On Mon, Nov 30, 2020 at 14:45:18 -0500, Luna Xu wrote: > > > Hi developers, > > > > Hi, > > > > [...] > > > > > When I start a VM with the qemu command I can specify the -no-shutdown > > > flag so that my qemu process doesn't quit even if I shutdown the VM > > > from the inside (issue shutdown or halt command inside VM). The VM is > > > shutdown but the qemu process is not killed so that the jobs I > > > submitted before for example backup (drive-backup) can continue to run > > > even after VM gets shutdown by the user, and I'm able to check the > > > status of the job through qmp commands (by communicating with the qemu > > > process). > > > > As I've noted in my reply on libvirt-users, we currently don't support > > this on the shutdown part of the VM lifecycle. > > > > On the startup part you can use VIR_DOMAIN_START_PAUSED for a VM which > > didn't yet excecute any guest instructions. > > > > > Is anyone aware of any project that is currently implementing this > > > functionality? Thanks for reading. > > > > I don't think that there's anybody working on it. > > > > If you are interested in implementing the feature the idea would be to > > add a 'pause' action for pause along with a > > new 'virDomainPausedReason' such as VIR_DOMAIN_PAUSED_POWEROFF and > > > implement the lifecycle transition from PAUSED -> RUNNING via > > qemuDomainResume, which would in this case have to issue a > > 'system-reset' qmp command and emit the appropriate events. > > This is weird, the domain was paused just before being shut down so > resuming it should logically just resume the paused process and finish > the shutdown. On the other hand, domain shutdown as a result of calling > resume is weird too. > > So how about just adding a check to qemuDomainResume to report an error > if you try to resume a domain which was paused for shutdown? Note we have a virDomainReset() API that does a full machine/cpu reset, so if we're in the puased state after shutdown, then a reset followed by a resume is a valid way to boot the guest OS fresh. This is in fact how we fake graceful reboots. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: qemu no-shutdown feature in libvirt
Hi Daniel, Thanks for replying. Yes a reset will do if the VM gets restarted. In this case, it makes sense to just need to implement the pause function on shutdown in libvirt for qemu. Thanks for pointing out. Thanks, Luna On Tue, Dec 1, 2020 at 12:53 PM Daniel P. Berrangé wrote: > > On Tue, Dec 01, 2020 at 05:09:26PM +0100, Jiri Denemark wrote: > > On Tue, Dec 01, 2020 at 15:34:05 +0100, Peter Krempa wrote: > > > On Mon, Nov 30, 2020 at 14:45:18 -0500, Luna Xu wrote: > > > > Hi developers, > > > > > > Hi, > > > > > > [...] > > > > > > > When I start a VM with the qemu command I can specify the -no-shutdown > > > > flag so that my qemu process doesn't quit even if I shutdown the VM > > > > from the inside (issue shutdown or halt command inside VM). The VM is > > > > shutdown but the qemu process is not killed so that the jobs I > > > > submitted before for example backup (drive-backup) can continue to run > > > > even after VM gets shutdown by the user, and I'm able to check the > > > > status of the job through qmp commands (by communicating with the qemu > > > > process). > > > > > > As I've noted in my reply on libvirt-users, we currently don't support > > > this on the shutdown part of the VM lifecycle. > > > > > > On the startup part you can use VIR_DOMAIN_START_PAUSED for a VM which > > > didn't yet excecute any guest instructions. > > > > > > > Is anyone aware of any project that is currently implementing this > > > > functionality? Thanks for reading. > > > > > > I don't think that there's anybody working on it. > > > > > > If you are interested in implementing the feature the idea would be to > > > add a 'pause' action for pause along with a > > > new 'virDomainPausedReason' such as VIR_DOMAIN_PAUSED_POWEROFF and > > > > > implement the lifecycle transition from PAUSED -> RUNNING via > > > qemuDomainResume, which would in this case have to issue a > > > 'system-reset' qmp command and emit the appropriate events. > > > > This is weird, the domain was paused just before being shut down so > > resuming it should logically just resume the paused process and finish > > the shutdown. On the other hand, domain shutdown as a result of calling > > resume is weird too. > > > > So how about just adding a check to qemuDomainResume to report an error > > if you try to resume a domain which was paused for shutdown? > > Note we have a virDomainReset() API that does a full machine/cpu reset, > so if we're in the puased state after shutdown, then a reset followed > by a resume is a valid way to boot the guest OS fresh. This is in fact > how we fake graceful reboots. > > Regards, > Daniel > -- > |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o-https://fstop138.berrange.com :| > |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| >
Re: qemu no-shutdown feature in libvirt
On Tue, Dec 01, 2020 at 17:53:30 +, Daniel P. Berrangé wrote: > On Tue, Dec 01, 2020 at 05:09:26PM +0100, Jiri Denemark wrote: > > On Tue, Dec 01, 2020 at 15:34:05 +0100, Peter Krempa wrote: > > > On Mon, Nov 30, 2020 at 14:45:18 -0500, Luna Xu wrote: > > > > Hi developers, > > > > > > Hi, > > > > > > [...] > > > > > > > When I start a VM with the qemu command I can specify the -no-shutdown > > > > flag so that my qemu process doesn't quit even if I shutdown the VM > > > > from the inside (issue shutdown or halt command inside VM). The VM is > > > > shutdown but the qemu process is not killed so that the jobs I > > > > submitted before for example backup (drive-backup) can continue to run > > > > even after VM gets shutdown by the user, and I'm able to check the > > > > status of the job through qmp commands (by communicating with the qemu > > > > process). > > > > > > As I've noted in my reply on libvirt-users, we currently don't support > > > this on the shutdown part of the VM lifecycle. > > > > > > On the startup part you can use VIR_DOMAIN_START_PAUSED for a VM which > > > didn't yet excecute any guest instructions. > > > > > > > Is anyone aware of any project that is currently implementing this > > > > functionality? Thanks for reading. > > > > > > I don't think that there's anybody working on it. > > > > > > If you are interested in implementing the feature the idea would be to > > > add a 'pause' action for pause along with a > > > new 'virDomainPausedReason' such as VIR_DOMAIN_PAUSED_POWEROFF and > > > > > implement the lifecycle transition from PAUSED -> RUNNING via > > > qemuDomainResume, which would in this case have to issue a > > > 'system-reset' qmp command and emit the appropriate events. > > > > This is weird, the domain was paused just before being shut down so > > resuming it should logically just resume the paused process and finish > > the shutdown. On the other hand, domain shutdown as a result of calling > > resume is weird too. > > > > So how about just adding a check to qemuDomainResume to report an error > > if you try to resume a domain which was paused for shutdown? > > Note we have a virDomainReset() API that does a full machine/cpu reset, > so if we're in the puased state after shutdown, then a reset followed > by a resume is a valid way to boot the guest OS fresh. This is in fact > how we fake graceful reboots. Sure, I just didn't want virDomainResume to automagically do the reset part. I guess you're suggesting that we should not report an error because virDomainReset would not change the domain state (or would it?) and thus reporting an error would block this scenario. So in that case, we should not report any error and in just try to resume the CPUs. In other words, this would mean no change is really needed in qemuDomainResume. Jirka
Re: qemu no-shutdown feature in libvirt
On Tue, Dec 01, 2020 at 07:38:51PM +0100, Jiri Denemark wrote: > On Tue, Dec 01, 2020 at 17:53:30 +, Daniel P. Berrangé wrote: > > On Tue, Dec 01, 2020 at 05:09:26PM +0100, Jiri Denemark wrote: > > > On Tue, Dec 01, 2020 at 15:34:05 +0100, Peter Krempa wrote: > > > > On Mon, Nov 30, 2020 at 14:45:18 -0500, Luna Xu wrote: > > > > > Hi developers, > > > > > > > > Hi, > > > > > > > > [...] > > > > > > > > > When I start a VM with the qemu command I can specify the -no-shutdown > > > > > flag so that my qemu process doesn't quit even if I shutdown the VM > > > > > from the inside (issue shutdown or halt command inside VM). The VM is > > > > > shutdown but the qemu process is not killed so that the jobs I > > > > > submitted before for example backup (drive-backup) can continue to run > > > > > even after VM gets shutdown by the user, and I'm able to check the > > > > > status of the job through qmp commands (by communicating with the qemu > > > > > process). > > > > > > > > As I've noted in my reply on libvirt-users, we currently don't support > > > > this on the shutdown part of the VM lifecycle. > > > > > > > > On the startup part you can use VIR_DOMAIN_START_PAUSED for a VM which > > > > didn't yet excecute any guest instructions. > > > > > > > > > Is anyone aware of any project that is currently implementing this > > > > > functionality? Thanks for reading. > > > > > > > > I don't think that there's anybody working on it. > > > > > > > > If you are interested in implementing the feature the idea would be to > > > > add a 'pause' action for pause along with a > > > > new 'virDomainPausedReason' such as VIR_DOMAIN_PAUSED_POWEROFF and > > > > > > > implement the lifecycle transition from PAUSED -> RUNNING via > > > > qemuDomainResume, which would in this case have to issue a > > > > 'system-reset' qmp command and emit the appropriate events. > > > > > > This is weird, the domain was paused just before being shut down so > > > resuming it should logically just resume the paused process and finish > > > the shutdown. On the other hand, domain shutdown as a result of calling > > > resume is weird too. > > > > > > So how about just adding a check to qemuDomainResume to report an error > > > if you try to resume a domain which was paused for shutdown? > > > > Note we have a virDomainReset() API that does a full machine/cpu reset, > > so if we're in the puased state after shutdown, then a reset followed > > by a resume is a valid way to boot the guest OS fresh. This is in fact > > how we fake graceful reboots. > > Sure, I just didn't want virDomainResume to automagically do the reset > part. I guess you're suggesting that we should not report an error > because virDomainReset would not change the domain state (or would it?) > and thus reporting an error would block this scenario. So in that case, > we should not report any error and in just try to resume the CPUs. In > other words, this would mean no change is really needed in > qemuDomainResume. We should be in the "SHUTDOWN" state when paused at the end of shutdown. Normally you'd never see this stsate as we immediately go into SHUTOFF. A virDomainReset should likely transition back to "RUNNING", but "PAUSED". So we can block on the state in virDomainResume if we're in SHUTDOWN. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: qemu no-shutdown feature in libvirt
On Tue, Dec 01, 2020 at 03:34:05PM +0100, Peter Krempa wrote: On Mon, Nov 30, 2020 at 14:45:18 -0500, Luna Xu wrote: Hi developers, Hi, [...] When I start a VM with the qemu command I can specify the -no-shutdown flag so that my qemu process doesn't quit even if I shutdown the VM from the inside (issue shutdown or halt command inside VM). The VM is shutdown but the qemu process is not killed so that the jobs I submitted before for example backup (drive-backup) can continue to run even after VM gets shutdown by the user, and I'm able to check the status of the job through qmp commands (by communicating with the qemu process). As I've noted in my reply on libvirt-users, we currently don't support this on the shutdown part of the VM lifecycle. On the startup part you can use VIR_DOMAIN_START_PAUSED for a VM which didn't yet excecute any guest instructions. Is anyone aware of any project that is currently implementing this functionality? Thanks for reading. I don't think that there's anybody working on it. If you are interested in implementing the feature the idea would be to add a 'pause' action for pause along with a new 'virDomainPausedReason' such as VIR_DOMAIN_PAUSED_POWEROFF and implement the lifecycle transition from PAUSED -> RUNNING via qemuDomainResume, which would in this case have to issue a 'system-reset' qmp command and emit the appropriate events. Couldn't this mechanism be used to add support for non-transient jobs on non-transient domains? The way I read the email that's exactly what the request is about. And that way libvirt APIs could be used for the block jobs without needing to mess with the domain definition. Maybe we don't even need to expose . signature.asc Description: PGP signature
