Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Stephen Stuart wrote: > I call this in /etc/rc.shutdown.local to shut all VMs down gracefully and > wait, perhaps a variant would help you. > > #!/bin/sh > > INTERVAL=5 > /usr/bin/killall -TERM bhyve > echo -n "/dev/vmm:" > while test -d /dev/vmm ; do > echo -n "" $(/bin/ls /dev/vmm | /usr/bin/wc -w) > /bin/sleep $INTERVAL > done > echo " 0" Thanks for reminding me about /etc/rc.shutdown.local. I've tested a solution today and it seems working. 1. Start apcupsd with --term-on-powerfail apcupsd_enable="YES" apcupsd_flags="--term-on-powerfail" 2. Put the following line into /etc/rc.shutdown.local test -f /var/run/powerfail && /usr/local/sbin/apcupsd --killpower Now, when the UPS power is exhausted, apcupsd starts the shutdown procedure and exits. The system shuts down without any haste, as it pleases, and at the very end "apcupsd --killpower" is called and poweroffs the UPS. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ signature.asc Description: PGP signature
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Victor Sudakov wrote:
>
> For the present I think I've found a workaround not directly related to
> vm-bhyve. I'm going to insert "service vm stop" into the doshutdown()
> procedure in /usr/local/etc/apcupsd/apccontrol before the actual
> ${SHUTDOWN}. It should give the VMs sufficient time to shutdown before
> the actual /sbin/shutdown is executed.
The workaround turned out useless: apcupsd sends the "kill power"
command to the UPS before calling the doshutdown() procedure.
F*ck, is there no correct, generally accepted way of powering off a busy
FreeBSD server? Why am I inventing the wheel?
> > If you know of a better way to configure apcupsd so that it powers down
> > the UPS only after all the guest OSes and other daemons are safely down,
> > I'd appreciate that too.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
signature.asc
Description: PGP signature
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
I call this in /etc/rc.shutdown.local to shut all VMs down gracefully and
wait, perhaps a variant would help you.
#!/bin/sh
INTERVAL=5
/usr/bin/killall -TERM bhyve
echo -n "/dev/vmm:"
while test -d /dev/vmm ; do
echo -n "" $(/bin/ls /dev/vmm | /usr/bin/wc -w)
/bin/sleep $INTERVAL
done
echo " 0"
Stephen
On Tue, Apr 23, 2019 at 11:53 PM Victor Sudakov wrote:
> Rodney W. Grimes wrote:
> > > >
> > > > I've noticed that it takes too long for Windows 2012 and 2016 servers
> > > > guests to shutdown when I issue "vm stopall," several minutes even.
> > > >
> > > > Does vm-bhyve provide a way to power them off ungracefully if they
> would
> > > > not stop within a predefined time?
> > > >
> > > > Or is there perhaps a way to tune the guest OSes to react more
> promptly
> > > > to an ACPI shutdown?
> > > >
> > > > My primary concern is that my UPS and "apcupsd --kill-on-powerfail"
> give
> > > > me about 30 seconds of grace time before the power supply is cut
> out. It
> > > > is not sufficient for the bhyve box to shutdown shutdown properly
> > > > because it waits for the guests to shutdown.
> > >
> > > For the present I think I've found a workaround not directly related to
> > > vm-bhyve. I'm going to insert "service vm stop" into the doshutdown()
> > > procedure in /usr/local/etc/apcupsd/apccontrol before the actual
> > > ${SHUTDOWN}. It should give the VMs sufficient time to shutdown before
> > > the actual /sbin/shutdown is executed.
> >
> > Make sure it does not do the waitall for pids or your going to
> > sit here waiting for that to complete for a long time, ie until
> > all vm's go to power off.
>
> In fact, this is my intention to sit here waiting for all VM's to go power
> off.
>
> I can configure apcupsd to start the doshutdown() routine when there is
> enough power in the UPS, like maybe for 20-30 minutes.
>
> > > > If you know of a better way to configure apcupsd so that it powers
> down
> > > > the UPS only after all the guest OSes and other daemons are safely
> down,
> > > > I'd appreciate that too.
> > >
> > > Do you think the above hack would have any negative effects?
> >
> > I think this is a reasonable approach given the current situation.
> >
>
> --
> Victor Sudakov, VAS4-RIPE, VAS47-RIPN
> 2:5005/49@fidonet http://vas.tomsk.ru/
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Rodney W. Grimes wrote:
> > >
> > > I've noticed that it takes too long for Windows 2012 and 2016 servers
> > > guests to shutdown when I issue "vm stopall," several minutes even.
> > >
> > > Does vm-bhyve provide a way to power them off ungracefully if they would
> > > not stop within a predefined time?
> > >
> > > Or is there perhaps a way to tune the guest OSes to react more promptly
> > > to an ACPI shutdown?
> > >
> > > My primary concern is that my UPS and "apcupsd --kill-on-powerfail" give
> > > me about 30 seconds of grace time before the power supply is cut out. It
> > > is not sufficient for the bhyve box to shutdown shutdown properly
> > > because it waits for the guests to shutdown.
> >
> > For the present I think I've found a workaround not directly related to
> > vm-bhyve. I'm going to insert "service vm stop" into the doshutdown()
> > procedure in /usr/local/etc/apcupsd/apccontrol before the actual
> > ${SHUTDOWN}. It should give the VMs sufficient time to shutdown before
> > the actual /sbin/shutdown is executed.
>
> Make sure it does not do the waitall for pids or your going to
> sit here waiting for that to complete for a long time, ie until
> all vm's go to power off.
In fact, this is my intention to sit here waiting for all VM's to go power off.
I can configure apcupsd to start the doshutdown() routine when there is
enough power in the UPS, like maybe for 20-30 minutes.
> > > If you know of a better way to configure apcupsd so that it powers down
> > > the UPS only after all the guest OSes and other daemons are safely down,
> > > I'd appreciate that too.
> >
> > Do you think the above hack would have any negative effects?
>
> I think this is a reasonable approach given the current situation.
>
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
signature.asc
Description: PGP signature
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
> Victor Sudakov wrote:
> >
> > I've noticed that it takes too long for Windows 2012 and 2016 servers
> > guests to shutdown when I issue "vm stopall," several minutes even.
> >
> > Does vm-bhyve provide a way to power them off ungracefully if they would
> > not stop within a predefined time?
> >
> > Or is there perhaps a way to tune the guest OSes to react more promptly
> > to an ACPI shutdown?
> >
> > My primary concern is that my UPS and "apcupsd --kill-on-powerfail" give
> > me about 30 seconds of grace time before the power supply is cut out. It
> > is not sufficient for the bhyve box to shutdown shutdown properly
> > because it waits for the guests to shutdown.
>
> For the present I think I've found a workaround not directly related to
> vm-bhyve. I'm going to insert "service vm stop" into the doshutdown()
> procedure in /usr/local/etc/apcupsd/apccontrol before the actual
> ${SHUTDOWN}. It should give the VMs sufficient time to shutdown before
> the actual /sbin/shutdown is executed.
Make sure it does not do the waitall for pids or your going to
sit here waiting for that to complete for a long time, ie until
all vm's go to power off.
> >
> > If you know of a better way to configure apcupsd so that it powers down
> > the UPS only after all the guest OSes and other daemons are safely down,
> > I'd appreciate that too.
>
> Do you think the above hack would have any negative effects?
I think this is a reasonable approach given the current situation.
> --
> Victor Sudakov, VAS4-RIPE, VAS47-RIPN
--
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Victor Sudakov wrote:
>
> I've noticed that it takes too long for Windows 2012 and 2016 servers
> guests to shutdown when I issue "vm stopall," several minutes even.
>
> Does vm-bhyve provide a way to power them off ungracefully if they would
> not stop within a predefined time?
>
> Or is there perhaps a way to tune the guest OSes to react more promptly
> to an ACPI shutdown?
>
> My primary concern is that my UPS and "apcupsd --kill-on-powerfail" give
> me about 30 seconds of grace time before the power supply is cut out. It
> is not sufficient for the bhyve box to shutdown shutdown properly
> because it waits for the guests to shutdown.
For the present I think I've found a workaround not directly related to
vm-bhyve. I'm going to insert "service vm stop" into the doshutdown()
procedure in /usr/local/etc/apcupsd/apccontrol before the actual
${SHUTDOWN}. It should give the VMs sufficient time to shutdown before
the actual /sbin/shutdown is executed.
>
> If you know of a better way to configure apcupsd so that it powers down
> the UPS only after all the guest OSes and other daemons are safely down,
> I'd appreciate that too.
Do you think the above hack would have any negative effects?
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
signature.asc
Description: PGP signature
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
> On Apr 22, 2019, at 21:13, Victor Sudakov wrote:
>
> Paul Vixie wrote:
>>
>> Victor Sudakov wrote on 2019-04-22 19:43:
>> ...
And the implementation is pretty brutal:
# 'vm stopall'
# stop all bhyve instances
# note this will also stop instances not started by vm-bhyve
#
core::stopall(){
local _pids=$(pgrep -f 'bhyve:')
echo "Shutting down all bhyve virtual machines"
killall bhyve
sleep 1
killall bhyve
wait_for_pids ${_pids}
}
>>
>> yow.
Eew no that is painful to read!
>
> To be sure, I was unable to find the above code (as is) in
> /usr/local/lib/vm-bhyve/vm-* (the vm-bhyve port 1.3.0). It may be that
> something more intelligent is happening in a more recent version, like a
> sequential shutdown. However, "kill $pid; sleep 1; kill $pid" seems to
> be still present.
>
>>
I wonder what the effect of the second kill is,
that seems odd.
>>>
>>> Indeed.
>>
>> the first killall will cause each client OS to see a soft shutdown
>> signal. the sleep 1 gives them some time to flush their buffers. the
>> second killall says, time's up, just stop.
>>
>> i think this is worse than brutal, it's wrong. consider freebsd's own
>> work flow when trying to comply with the first soft shutdown it got:
>>
>> https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220
>>
>> this has bitten me more than once, because using "pageins" as a proxy
>> for "my server processes are busy trying to synchronize their user mode
>> state" is inaccurate. i think _any_ continuing I/O should be reason to
>> wait the full 60 seconds.
>
> Would it be beneficial to just hack /usr/local/lib/vm-bhyve/vm-* ?
>>
>> and so i think the "sleep 1" above should be a "sleep 65".
I would echo this and say it should probably be done in a way that you can have
a sliding window, some servers and services are not very fault tolerant on
their own. The example that springs to mind for me is the busy AD domain
controller I manage. It takes 15 mins to flush the disk buffer, if I kill it
before the buffer flushes I will have a bad day as my domain at best loses a
few transactions at worst is corrupted.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Victor Sudakov wrote on 2019-04-22 21:13: ... sleep 1 ... i think this is worse than brutal, it's wrong. consider freebsd's own work flow when trying to comply with the first soft shutdown it got: https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220 this has bitten me more than once, because using "pageins" as a proxy for "my server processes are busy trying to synchronize their user mode state" is inaccurate. i think _any_ continuing I/O should be reason to wait the full 60 seconds. Would it be beneficial to just hack /usr/local/lib/vm-bhyve/vm-* ? no, i really think the hypervisor should add i/o monitoring, so that if the soft shutdown signal is sent, the client kernel is allowed to keep running until either a really long timeout (60 seconds or more), but if it becomes quiet, then the hypervisor can exit sooner. i agree with this. Will you please support the bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237479 done. -- P Vixie ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Victor Sudakov wrote on 2019-04-22 19:43:
...
>> And the implementation is pretty brutal:
>> # 'vm stopall'
>> # stop all bhyve instances
>> # note this will also stop instances not started by vm-bhyve #
>> core::stopall(){
>> local _pids=$(pgrep -f 'bhyve:')
>>
>> echo "Shutting down all bhyve virtual machines"
>> killall bhyve
>> sleep 1
>> killall bhyve
>> wait_for_pids ${_pids}
>> }
> yow.
>>
>> I wonder what the effect of the second kill is, that seems odd.
>
> Indeed.
> the first killall will cause each client OS to see a soft shutdown signal.
> the sleep 1 gives them some time to flush their buffers. the second killall
> says, time's up, just stop.
Both of these killall commands send a soft shutdown signal and I've never seen
an instance in my own testing where the second has caused an instant poweroff.
I've just double checked this, and a FreeBSD guest stopped with the existing
code fully shuts itself down, ending with "acpi0: Powering system off"
The main reason for having both is that in my initial testing, I could not get
Windows to respond to a single SIGINT. 100% of the time it would simply act
like nothing had happened. Sending two however triggered a shutdown. I only
have a single test machine though so maybe it's something strange about my
particular system that no one else experiences...
Matt
> i think this is worse than brutal, it's wrong. consider freebsd's own work
> flow when trying to comply with the first soft shutdown it got:
> https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220
> this has bitten me more than once, because using "pageins" as a proxy for "my
> server processes are busy trying to synchronize their user mode state" is
> inaccurate. i think _any_ continuing I/O should > be reason to wait the full
> 60 seconds.
> and so i think the "sleep 1" above should be a "sleep 65".
> What is needed in vm-bhyve is the feature that if ACPI does not stop
> the guest for a predefined period of time, the guest is powered off.
> i agree with this.
--
> P Vixie
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
-- Start of PGP signed section.
> Paul Vixie wrote:
> >
> > Victor Sudakov wrote on 2019-04-22 19:43:
> > ...
> > >> And the implementation is pretty brutal:
> > >> # 'vm stopall'
> > >> # stop all bhyve instances
> > >> # note this will also stop instances not started by vm-bhyve
> > >> #
> > >> core::stopall(){
> > >> local _pids=$(pgrep -f 'bhyve:')
> > >>
> > >> echo "Shutting down all bhyve virtual machines"
> > >> killall bhyve
> > >> sleep 1
> > >> killall bhyve
> > >> wait_for_pids ${_pids}
> > >> }
> >
> > yow.
>
> To be sure, I was unable to find the above code (as is) in
> /usr/local/lib/vm-bhyve/vm-* (the vm-bhyve port 1.3.0). It may be that
> something more intelligent is happening in a more recent version, like a
> sequential shutdown. However, "kill $pid; sleep 1; kill $pid" seems to
> be still present.
I probably pulled that from old code, pulled from:
vm-bhyve-1.2.3 Management system for bhyve virtual machines
>
> >
> > >>
> > >> I wonder what the effect of the second kill is,
> > >> that seems odd.
> > >
> > > Indeed.
> >
> > the first killall will cause each client OS to see a soft shutdown
> > signal. the sleep 1 gives them some time to flush their buffers. the
> > second killall says, time's up, just stop.
> >
> > i think this is worse than brutal, it's wrong. consider freebsd's own
> > work flow when trying to comply with the first soft shutdown it got:
> >
> > https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220
> >
> > this has bitten me more than once, because using "pageins" as a proxy
> > for "my server processes are busy trying to synchronize their user mode
> > state" is inaccurate. i think _any_ continuing I/O should be reason to
> > wait the full 60 seconds.
>
> Would it be beneficial to just hack /usr/local/lib/vm-bhyve/vm-* ?
One can always hack experiments, my vm-bhyve is fairly diverged
from the release bits as I have done stuff to it so that it
better meets my needs. Most of that is not submittable as upstream
changes, some I should really sort out and try to push up.
Some is due to local changes to bhyve that are not mainstream
and thus are not (yet) applicable.
> > and so i think the "sleep 1" above should be a "sleep 65".
> >
> > > What is needed in vm-bhyve is the feature that if ACPI does not stop the
> > > guest for a predefined period of time, the guest is powered off.
> >
> > i agree with this.
>
> Will you please support the bug report:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237479
More powerfull would be if we could come up with some
patches against 1.3.0 that effected some of the changes
we desire. And some more investigation as to just how
the guests are handling this ACPI shutdown event. What
might be wrong for FreeBSD might be right for windows?
Does the ACPI spec saying anything about hitting the
power down button twice within 1 second for example?
> Victor Sudakov, VAS4-RIPE, VAS47-RIPN
--
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
> Victor Sudakov wrote on 2019-04-22 19:43:
> ...
> >> And the implementation is pretty brutal:
> >> # 'vm stopall'
> >> # stop all bhyve instances
> >> # note this will also stop instances not started by vm-bhyve
> >> #
> >> core::stopall(){
> >> local _pids=$(pgrep -f 'bhyve:')
> >>
> >> echo "Shutting down all bhyve virtual machines"
> >> killall bhyve
> >> sleep 1
> >> killall bhyve
> >> wait_for_pids ${_pids}
> >> }
>
> yow.
>
> >>
> >> I wonder what the effect of the second kill is,
> >> that seems odd.
> >
> > Indeed.
>
> the first killall will cause each client OS to see a soft shutdown
> signal. the sleep 1 gives them some time to flush their buffers. the
> second killall says, time's up, just stop.
>
> i think this is worse than brutal, it's wrong. consider freebsd's own
> work flow when trying to comply with the first soft shutdown it got:
>
> https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220
Interesting, more interesting would be to dig out the
path that the system takes when it gets teh ACPI shutdown
event. That is not going to end up in sbin/reboot is it?
Doesnt that run from within init itself?
I find in the init man page this:
The init utility will terminate all possible processes (again, it will
not wait for deadlocked processes) and reboot the machine if sent the
interrupt (INT) signal, i.e. ``kill -INT 1''. This is useful for
shutting the machine down cleanly from inside the kernel or from X when
the machine appears to be hung.
So my guess is that sending an ACPI shutdown event to the VM ends up
sending a -INT to init. Looking inside init, it seems to end up
execing a shell runing /etc/rc.shutdown.
I do not know if the ACPI event is then blocked in the kernel so
the second one is pointless, but I believe once we enter into
-INT processing that signal is ignored, so infact this second
signal is actaully OK. But I am not 100% sure on this, yet.
> this has bitten me more than once, because using "pageins" as a proxy
> for "my server processes are busy trying to synchronize their user mode
> state" is inaccurate. i think _any_ continuing I/O should be reason to
> wait the full 60 seconds.
>
> and so i think the "sleep 1" above should be a "sleep 65".
I think the sleep and the second signal are near pointless. If
we have a race in ACPI processing we need to fix that.
> > What is needed in vm-bhyve is the feature that if ACPI does not stop the
> > guest for a predefined period of time, the guest is powered off.
>
> i agree with this.
There is a PR up,
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237479
please add comments there if you feel so inclined.
> --
> P Vixie
--
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Paul Vixie wrote:
>
> Victor Sudakov wrote on 2019-04-22 19:43:
> ...
> >> And the implementation is pretty brutal:
> >> # 'vm stopall'
> >> # stop all bhyve instances
> >> # note this will also stop instances not started by vm-bhyve
> >> #
> >> core::stopall(){
> >> local _pids=$(pgrep -f 'bhyve:')
> >>
> >> echo "Shutting down all bhyve virtual machines"
> >> killall bhyve
> >> sleep 1
> >> killall bhyve
> >> wait_for_pids ${_pids}
> >> }
>
> yow.
To be sure, I was unable to find the above code (as is) in
/usr/local/lib/vm-bhyve/vm-* (the vm-bhyve port 1.3.0). It may be that
something more intelligent is happening in a more recent version, like a
sequential shutdown. However, "kill $pid; sleep 1; kill $pid" seems to
be still present.
>
> >>
> >> I wonder what the effect of the second kill is,
> >> that seems odd.
> >
> > Indeed.
>
> the first killall will cause each client OS to see a soft shutdown
> signal. the sleep 1 gives them some time to flush their buffers. the
> second killall says, time's up, just stop.
>
> i think this is worse than brutal, it's wrong. consider freebsd's own
> work flow when trying to comply with the first soft shutdown it got:
>
> https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220
>
> this has bitten me more than once, because using "pageins" as a proxy
> for "my server processes are busy trying to synchronize their user mode
> state" is inaccurate. i think _any_ continuing I/O should be reason to
> wait the full 60 seconds.
Would it be beneficial to just hack /usr/local/lib/vm-bhyve/vm-* ?
>
> and so i think the "sleep 1" above should be a "sleep 65".
>
> > What is needed in vm-bhyve is the feature that if ACPI does not stop the
> > guest for a predefined period of time, the guest is powered off.
>
> i agree with this.
Will you please support the bug report:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237479
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
signature.asc
Description: PGP signature
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Victor Sudakov wrote on 2019-04-22 19:43:
...
And the implementation is pretty brutal:
# 'vm stopall'
# stop all bhyve instances
# note this will also stop instances not started by vm-bhyve
#
core::stopall(){
local _pids=$(pgrep -f 'bhyve:')
echo "Shutting down all bhyve virtual machines"
killall bhyve
sleep 1
killall bhyve
wait_for_pids ${_pids}
}
yow.
I wonder what the effect of the second kill is,
that seems odd.
Indeed.
the first killall will cause each client OS to see a soft shutdown
signal. the sleep 1 gives them some time to flush their buffers. the
second killall says, time's up, just stop.
i think this is worse than brutal, it's wrong. consider freebsd's own
work flow when trying to comply with the first soft shutdown it got:
https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220
this has bitten me more than once, because using "pageins" as a proxy
for "my server processes are busy trying to synchronize their user mode
state" is inaccurate. i think _any_ continuing I/O should be reason to
wait the full 60 seconds.
and so i think the "sleep 1" above should be a "sleep 65".
What is needed in vm-bhyve is the feature that if ACPI does not stop the
guest for a predefined period of time, the guest is powered off.
i agree with this.
--
P Vixie
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Victor Sudakov wrote: [dd] > > > > It may be possile to adjust vm_delay to 0 and have that be better, > > > > though I have not locked at the code. You may also wish to discuss > > > > the issue with the vm-bhyve maintainer and maybe a "lights out" > > > > procedure needs to be added. > > What is needed in vm-bhyve is the feature that if ACPI does not stop the > guest for a predefined period of time, the guest is powered off. I've created a feature request https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237479 please support me for adding yourself to the CC list, or commenting. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ signature.asc Description: PGP signature
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Rodney W. Grimes wrote:
[dd]
> > > That is more likely your problem in that your sending these acpi
> > > shutdown requests one at a time, and they should be broadcast in
> > > the "power going out" case.
> >
> > Whence is the idea that "vm stopall" does a sequential shutdown? What sense
> > would that make?
>
> Well I am not sure that vm-bhyve does this, but esxi certainly does,
> and it is a royal PITA to deal with sometimes. It does make sense
> in some aspects, do not want the database server going offline before
> all the clients go down do we? Kinda makes for a bunch of nonsense
> errors logged due to missing server. I kinda like my virtual routers
> and firewalls to keep going tell the end too.
>
> This is all YMMV situations though.
OK, you have convinced me, a predictable sequential shutdown may make
sense sometimes. Anyway, it's not there im vm-bhyve so it's not the
reason for the slow shutdown.
>
> > A sequential startup does make sense but a sequential shutdown? Useless
> > I think. The man page says that
>
> For you perhaps useless, but that rarely rules out that there may be
> a totally valid and usefull case.
>
> > stopall
> > Stop all running virtual machines. This sends a stop command
> > to
> > all bhyve(8) instances, regardless of whether they were
> > starting
> > using vm or not.
>
> And the implementation is pretty brutal:
> # 'vm stopall'
> # stop all bhyve instances
> # note this will also stop instances not started by vm-bhyve
> #
> core::stopall(){
> local _pids=$(pgrep -f 'bhyve:')
>
> echo "Shutting down all bhyve virtual machines"
> killall bhyve
> sleep 1
> killall bhyve
> wait_for_pids ${_pids}
> }
>
> I wonder what the effect of the second kill is,
> that seems odd.
Indeed.
> Almost like you might cause more
> issues than you solve as now you already have a
> vm in what should be ACPI shutdown process.
>
> Also this killall operation probably puts a high stress
> on disk i/o as you just woke up and made all the vm's
> get busy all at once and your going to basically thrash
> on every single resource they are using (yet another reason
> you may actually want to serialize these shutdown operations.)
You are right.
>
> IIRC windows, especially newer ones, do a boat load of work
> on a shutdown unless they are told to shutdown quickly. Ie
> they even try to apply pending updates and such, that could
> be part of why you see windows vm's lagging around.
Do you know how to configure Windows for an unconditional ACPI shutdown?
Last time I stopped my Windows guests, they stopped pretty quickly. But
sometimes it just takes them forever to stop. Maybe it was giving a
shutdown warning to the users or something. Or maybe it was this issue:
https://serverfault.com/questions/871792/acpi-shutdown-does-not-always-work-on-a-windows-server-virtual-machine
> You may also want to: Disable Clear page file on shutdown
> that is a windows thing, if you have a huge page file that
> can do a LOT of io, if you have a few windows vm's on the
> same spindle and try to stop them all at once your going to
> trash that disk for much longer than you need.
Last time I checked on the Windows 2012 and 2016 servers, the
ClearPageFileAtShutdown setting was 0x0. I think it is the default.
> > > It may be possile to adjust vm_delay to 0 and have that be better,
> > > though I have not locked at the code. You may also wish to discuss
> > > the issue with the vm-bhyve maintainer and maybe a "lights out"
> > > procedure needs to be added.
What is needed in vm-bhyve is the feature that if ACPI does not stop the
guest for a predefined period of time, the guest is powered off.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
signature.asc
Description: PGP signature
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
-- Start of PGP signed section.
> Rodney W. Grimes wrote:
> > -- Start of PGP signed section.
> > > Jason Tubnor wrote:
> > > > On Sun., 21 Apr. 2019, 2:51 pm Victor Sudakov,
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > Does vm-bhyve provide a way to power them off ungracefully if they
> > > > > would
> > > > > not stop within a predefined time?
> > > > >
> > > >
> > > > You'd have to do your own checking but to force an instant power off of
> > > > a
> > > > guest simply execute:
> > > >
> > > > vm poweroff {guest}
> > >
> > > I don't need an *instant* poweroff, I need a poweroff only if a VM would
> > > not stop gracefully within a predefined time.
> > >
> > > Of course this should not be done manually but from the rc.d/vm script
> > > on shutdown.
> >
> > vm shutdown {guest} &
> > (sleep N && vm poweroff {guest}) &
> >
> > A bit hackish, but that should work, if the guest has
> > shutdown by the time you get to the poweroff an error
> > might occur. This spawns a few processes, but is
> > non blocking and highly parallel.
>
> Thank you, I might give it a try though it would involve hacking the
> rc.d/vm script.
>
> >
> > One thing I noted, you said rc.d/vm, so I am assuming your actually
> > using the vm-bhyve port,
>
> Yes, I thought that was pretty obvious from the Subject line.
Subject line disappears off the screen once a screen full of
email gets displayed :-)
> > in which case it has a sequential shutdown
> > with time delays between each guest. vm_delay specifically, and it
> > shuts vm's down in reverse order of the startup.
> >
> > That is more likely your problem in that your sending these acpi
> > shutdown requests one at a time, and they should be broadcast in
> > the "power going out" case.
>
> Whence is the idea that "vm stopall" does a sequential shutdown? What sense
> would that make?
Well I am not sure that vm-bhyve does this, but esxi certainly does,
and it is a royal PITA to deal with sometimes. It does make sense
in some aspects, do not want the database server going offline before
all the clients go down do we? Kinda makes for a bunch of nonsense
errors logged due to missing server. I kinda like my virtual routers
and firewalls to keep going tell the end too.
This is all YMMV situations though.
> A sequential startup does make sense but a sequential shutdown? Useless
> I think. The man page says that
For you perhaps useless, but that rarely rules out that there may be
a totally valid and usefull case.
> stopall
> Stop all running virtual machines. This sends a stop command to
> all bhyve(8) instances, regardless of whether they were starting
> using vm or not.
And the implementation is pretty brutal:
# 'vm stopall'
# stop all bhyve instances
# note this will also stop instances not started by vm-bhyve
#
core::stopall(){
local _pids=$(pgrep -f 'bhyve:')
echo "Shutting down all bhyve virtual machines"
killall bhyve
sleep 1
killall bhyve
wait_for_pids ${_pids}
}
I wonder what the effect of the second kill is,
that seems odd. Almost like you might cause more
issues than you solve as now you already have a
vm in what should be ACPI shutdown process.
Also this killall operation probably puts a high stress
on disk i/o as you just woke up and made all the vm's
get busy all at once and your going to basically thrash
on every single resource they are using (yet another reason
you may actually want to serialize these shutdown operations.)
IIRC windows, especially newer ones, do a boat load of work
on a shutdown unless they are told to shutdown quickly. Ie
they even try to apply pending updates and such, that could
be part of why you see windows vm's lagging around.
You may also want to: Disable Clear page file on shutdown
that is a windows thing, if you have a huge page file that
can do a LOT of io, if you have a few windows vm's on the
same spindle and try to stop them all at once your going to
trash that disk for much longer than you need.
> >
> > It may be possile to adjust vm_delay to 0 and have that be better,
> > though I have not locked at the code. You may also wish to discuss
> > the issue with the vm-bhyve maintainer and maybe a "lights out"
> > procedure needs to be added.
>
> Even if there is a single Windows 2012 or 2016 guest running, it takes
> the vm several minutes to transition to the "stopped" state.
You might nuke the wait_for_pids?
> Victor Sudakov, VAS4-RIPE, VAS47-RIPN
--
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Rodney W. Grimes wrote:
> -- Start of PGP signed section.
> > Jason Tubnor wrote:
> > > On Sun., 21 Apr. 2019, 2:51 pm Victor Sudakov,
> > > wrote:
> > >
> > > >
> > > >
> > > > Does vm-bhyve provide a way to power them off ungracefully if they would
> > > > not stop within a predefined time?
> > > >
> > >
> > > You'd have to do your own checking but to force an instant power off of a
> > > guest simply execute:
> > >
> > > vm poweroff {guest}
> >
> > I don't need an *instant* poweroff, I need a poweroff only if a VM would
> > not stop gracefully within a predefined time.
> >
> > Of course this should not be done manually but from the rc.d/vm script
> > on shutdown.
>
> vm shutdown {guest} &
> (sleep N && vm poweroff {guest}) &
>
> A bit hackish, but that should work, if the guest has
> shutdown by the time you get to the poweroff an error
> might occur. This spawns a few processes, but is
> non blocking and highly parallel.
Thank you, I might give it a try though it would involve hacking the
rc.d/vm script.
>
> One thing I noted, you said rc.d/vm, so I am assuming your actually
> using the vm-bhyve port,
Yes, I thought that was pretty obvious from the Subject line.
> in which case it has a sequential shutdown
> with time delays between each guest. vm_delay specifically, and it
> shuts vm's down in reverse order of the startup.
>
> That is more likely your problem in that your sending these acpi
> shutdown requests one at a time, and they should be broadcast in
> the "power going out" case.
Whence is the idea that "vm stopall" does a sequential shutdown? What sense
would that make?
A sequential startup does make sense but a sequential shutdown? Useless
I think. The man page says that
stopall
Stop all running virtual machines. This sends a stop command to
all bhyve(8) instances, regardless of whether they were starting
using vm or not.
>
> It may be possile to adjust vm_delay to 0 and have that be better,
> though I have not locked at the code. You may also wish to discuss
> the issue with the vm-bhyve maintainer and maybe a "lights out"
> procedure needs to be added.
Even if there is a single Windows 2012 or 2016 guest running, it takes
the vm several minutes to transition to the "stopped" state.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
signature.asc
Description: PGP signature
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
-- Start of PGP signed section.
> Jason Tubnor wrote:
> > On Sun., 21 Apr. 2019, 2:51 pm Victor Sudakov, wrote:
> >
> > >
> > >
> > > Does vm-bhyve provide a way to power them off ungracefully if they would
> > > not stop within a predefined time?
> > >
> >
> > You'd have to do your own checking but to force an instant power off of a
> > guest simply execute:
> >
> > vm poweroff {guest}
>
> I don't need an *instant* poweroff, I need a poweroff only if a VM would
> not stop gracefully within a predefined time.
>
> Of course this should not be done manually but from the rc.d/vm script
> on shutdown.
vm shutdown {guest} &
(sleep N && vm poweroff {guest}) &
A bit hackish, but that should work, if the guest has
shutdown by the time you get to the poweroff an error
might occur. This spawns a few processes, but is
non blocking and highly parallel.
One thing I noted, you said rc.d/vm, so I am assuming your actually
using the vm-bhyve port, in which case it has a sequential shutdown
with time delays between each guest. vm_delay specifically, and it
shuts vm's down in reverse order of the startup.
That is more likely your problem in that your sending these acpi
shutdown requests one at a time, and they should be broadcast in
the "power going out" case.
It may be possile to adjust vm_delay to 0 and have that be better,
though I have not locked at the code. You may also wish to discuss
the issue with the vm-bhyve maintainer and maybe a "lights out"
procedure needs to be added.
> Victor Sudakov, VAS4-RIPE, VAS47-RIPN
Regards,
--
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Jason Tubnor wrote:
> On Sun., 21 Apr. 2019, 2:51 pm Victor Sudakov, wrote:
>
> >
> >
> > Does vm-bhyve provide a way to power them off ungracefully if they would
> > not stop within a predefined time?
> >
>
> You'd have to do your own checking but to force an instant power off of a
> guest simply execute:
>
> vm poweroff {guest}
I don't need an *instant* poweroff, I need a poweroff only if a VM would
not stop gracefully within a predefined time.
Of course this should not be done manually but from the rc.d/vm script
on shutdown.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
signature.asc
Description: PGP signature
Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
On Sun., 21 Apr. 2019, 2:51 pm Victor Sudakov, wrote:
>
>
> Does vm-bhyve provide a way to power them off ungracefully if they would
> not stop within a predefined time?
>
You'd have to do your own checking but to force an instant power off of a
guest simply execute:
vm poweroff {guest}
Cheers.
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
[vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Dear Colleages, I've noticed that it takes too long for Windows 2012 and 2016 servers guests to shutdown when I issue "vm stopall," several minutes even. Does vm-bhyve provide a way to power them off ungracefully if they would not stop within a predefined time? Or is there perhaps a way to tune the guest OSes to react more promptly to an ACPI shutdown? My primary concern is that my UPS and "apcupsd --kill-on-powerfail" give me about 30 seconds of grace time before the power supply is cut out. It is not sufficient for the bhyve box to shutdown shutdown properly because it waits for the guests to shutdown. If you know of a better way to configure apcupsd so that it powers down the UPS only after all the guest OSes and other daemons are safely down, I'd appreciate that too. For some reason, apctest does not allow me to change the grace period, perhaps this model does not support it. And it would mean a race condition anyway. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ signature.asc Description: PGP signature
