Re: bhyve: Detecting that a guest kernel has booted

[Nick Wolff]

We actually have some people it looks like working on an updated UEFI and
have a CFT out in Updating "uefi-edk2-bhyve" thread in this mailing list.

I hope we can get bhyve-vm-goagent
 back in the tree and also
built for any other operating systems people need.

Thanks,

Nick Wolff


On Mon, Mar 11, 2019 at 8:21 PM Rodney W. Grimes <
freebsd-...@gndrsh.dnsmgr.net> wrote:

> > > This branch is 8930 commits behind tianocore:master.
> > > And depends on an external git belonging to Peter Grehan,
> >
> >   No, it isn't.
> >
> >   https://github.com/freebsd/uefi-edk2
> >
> >   Under FreeBSD project control since Oct 16, 2015.
>
> My appologies, the page does not clearly show who owns
> the git project and I wrongly assumed it was you.
>
> Though perhaps there is another edk2 some place
> that some people seem to have copies of?
>
> Is the source the same for the versions in
> people.freebsd.org/~grehan/?
>
>
> > later,
> > Peter.
> --
> Rod Grimes
> rgri...@freebsd.org
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Rodney W. Grimes]

> > This branch is 8930 commits behind tianocore:master.
> > And depends on an external git belonging to Peter Grehan,
> 
>   No, it isn't.
> 
>   https://github.com/freebsd/uefi-edk2
> 
>   Under FreeBSD project control since Oct 16, 2015.

My appologies, the page does not clearly show who owns
the git project and I wrongly assumed it was you.

Though perhaps there is another edk2 some place
that some people seem to have copies of?

Is the source the same for the versions in
people.freebsd.org/~grehan/?


> later,
> Peter.
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[grehan]

This branch is 8930 commits behind tianocore:master.
And depends on an external git belonging to Peter Grehan,


 No, it isn't.

 https://github.com/freebsd/uefi-edk2

 Under FreeBSD project control since Oct 16, 2015.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Rodney W. Grimes]

> Hi all,
> 
> > Am 11.03.2019 um 19:09 schrieb Rodney W. Grimes 
> > :
> > I have no love for them either, but until we get our UEFI updated
> > it is a rather sad state of affairs.
> 
> I routinely boot FreeBSD, Ubuntu and Windows 10 with
> UEFI, so I don?t see much that I would consider broken.

All 64 bit oses, afaik we can not boot anything 32 bit
with our current uefi, which is:
This branch is 8930 commits behind tianocore:master. 
And depends on an external git belonging to Peter Grehan,
who is now gone from the project.

> 
> Missing persistence of boot variables (correct term?) is
> the only thing that I know of - what else makes it a
> sad state, currently?

Our GOP code frequently blows up vnc clients is another
issue, not sure if that is in the uefi stuff, or if we
just have bad vnc layered on top of it.  It throws recs
that are out of bounds.

The embeded PXE code is also rather dated.

> Patrick

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Patrick M. Hausen]

Hi all,

> Am 11.03.2019 um 19:09 schrieb Rodney W. Grimes 
> :
> I have no love for them either, but until we get our UEFI updated
> it is a rather sad state of affairs.

I routinely boot FreeBSD, Ubuntu and Windows 10 with
UEFI, so I don’t see much that I would consider broken.

Missing persistence of boot variables (correct term?) is
the only thing that I know of - what else makes it a
sad state, currently?

Kind regards
Patrick
-- 
punkt.de GmbH   Internet - Dienstleistungen - Beratung
Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100
76133 Karlsruhe i...@punkt.de   http://punkt.de
AG Mannheim 108285  Gf: Juergen Egeling

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Patrick M. Hausen]

Hi all,

> Am 11.03.2019 um 18:04 schrieb Mark Raynsford via freebsd-virtualization 
> :
> Leaving aside userland monitoring tools such as Prometheus[0], is there
> any way to detect on the host that a guest kernel running in bhyve has
> booted? I'm assuming "booted" in this sense to mean "PID 1 has probably
> started". I'm guessing that there probably isn't, but I thought I'd
> better ask anyway. "Not booted" would mean something like "Is sitting
> at the Grub prompt doing nothing“.

The abandoned FreeNAS Corral release did have guest additions
that served as a simple health monitor for FreeBSD and Linux
guests.

They used virtio_console(4) and the last state of the project can
be found here:
https://github.com/freenas/freenas-vm-tools

They compile cleanly on FreeBSD 12 and I even made a port that
I submitted to iX Systems, but we all know what happened to
Corral, unfortunately.

Possibly they can serve as a starting point for a new project.

And then there seems to be a newer version, again from FreeNAS,
written in Go:
https://github.com/freenas/bhyve-vm-goagent

HTH,
Patrick

--
punkt.de GmbH   Internet - Dienstleistungen - Beratung
Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100
76133 Karlsruhe i...@punkt.de   http://punkt.de
AG Mannheim 108285  Gf: Juergen Egeling



signature.asc
Description: Message signed with OpenPGP

Re: bhyve: Detecting that a guest kernel has booted

[Shawn Webb]

On Mon, Mar 11, 2019 at 11:09:07AM -0700, Rodney W. Grimes wrote:
> > On Mon, Mar 11, 2019 at 10:58:55AM -0700, Rodney W. Grimes wrote:
> > > -- Start of PGP signed section.
> > > > On 2019-03-11T13:08:53 -0400
> > > > Shawn Webb  wrote:
> > > > >
> > > > > If your guest OS supports it, you could probably write two scripts 
> > > > > that
> > > > > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > > > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > > > > received.
> > > > > 
> > > > 
> > > > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> > > > one out of three of those supports it...
> > > > 
> > > > I suppose my other option would be to add (another) NFS mount in each
> > > > guest, and have them touch a file early in the init script (and
> > > > possibly touch a different file early in the shutdown script).
> > > 
> > > Well ICMP is in the kernel, and should be working as soon as the
> > > interface is up, long before you could do anything with NFS,
> > > so rather than the complexity above a simple ping would suffice.
> > 
> > Just a note: Windows systems disable inbound ICMP by default, but
> > inbound ICMP support can be enabled post-installation.
> > 
> > > There is also the phase of vmm(8) startup that when you are
> > > running bhyveload vs bhyve and iirc grubload vs bhyve, that
> > > can be detected.  vmbhyve does so and says you are in state
> > > looader when you do a vm list.
> > 
> > I would suggest using bhyve with UEFI. I wish a death upon bhyveload
> > and grub2-bhyve.
> 
> I have no love for them either, but until we get our UEFI updated
> it is a rather sad state of affairs.  If we could a) get on a modern
> version of ed2k, and b) bet the CSM fixed so that we could actually
> boot bios mode stuff with it and c) maybe make a port of seabios
> that could be used then we would be kicking some seriuos stuff!

If I'm able to run an internship this year like I did last year, I
might have my interns update our UEFI firmware to the latest release
and/or make a BSDL seabios port.

My employer was recently acquired, so I'm likely not going to run an
internship this year.

Thanks,

-- 
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:+1 443-546-8752
Tor+XMPP+OTR:latt...@is.a.hacker.sx
GPG Key ID:  0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE


signature.asc
Description: PGP signature

Re: bhyve: Detecting that a guest kernel has booted

[Rodney W. Grimes]

> On Mon, Mar 11, 2019 at 10:58:55AM -0700, Rodney W. Grimes wrote:
> > -- Start of PGP signed section.
> > > On 2019-03-11T13:08:53 -0400
> > > Shawn Webb  wrote:
> > > >
> > > > If your guest OS supports it, you could probably write two scripts that
> > > > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > > > received.
> > > > 
> > > 
> > > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> > > one out of three of those supports it...
> > > 
> > > I suppose my other option would be to add (another) NFS mount in each
> > > guest, and have them touch a file early in the init script (and
> > > possibly touch a different file early in the shutdown script).
> > 
> > Well ICMP is in the kernel, and should be working as soon as the
> > interface is up, long before you could do anything with NFS,
> > so rather than the complexity above a simple ping would suffice.
> 
> Just a note: Windows systems disable inbound ICMP by default, but
> inbound ICMP support can be enabled post-installation.
> 
> > There is also the phase of vmm(8) startup that when you are
> > running bhyveload vs bhyve and iirc grubload vs bhyve, that
> > can be detected.  vmbhyve does so and says you are in state
> > looader when you do a vm list.
> 
> I would suggest using bhyve with UEFI. I wish a death upon bhyveload
> and grub2-bhyve.

I have no love for them either, but until we get our UEFI updated
it is a rather sad state of affairs.  If we could a) get on a modern
version of ed2k, and b) bet the CSM fixed so that we could actually
boot bios mode stuff with it and c) maybe make a port of seabios
that could be used then we would be kicking some seriuos stuff!

> Thanks,
> Shawn Webb

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Shawn Webb]

On Mon, Mar 11, 2019 at 10:58:55AM -0700, Rodney W. Grimes wrote:
> -- Start of PGP signed section.
> > On 2019-03-11T13:08:53 -0400
> > Shawn Webb  wrote:
> > >
> > > If your guest OS supports it, you could probably write two scripts that
> > > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > > received.
> > > 
> > 
> > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> > one out of three of those supports it...
> > 
> > I suppose my other option would be to add (another) NFS mount in each
> > guest, and have them touch a file early in the init script (and
> > possibly touch a different file early in the shutdown script).
> 
> Well ICMP is in the kernel, and should be working as soon as the
> interface is up, long before you could do anything with NFS,
> so rather than the complexity above a simple ping would suffice.

Just a note: Windows systems disable inbound ICMP by default, but
inbound ICMP support can be enabled post-installation.

> There is also the phase of vmm(8) startup that when you are
> running bhyveload vs bhyve and iirc grubload vs bhyve, that
> can be detected.  vmbhyve does so and says you are in state
> looader when you do a vm list.

I would suggest using bhyve with UEFI. I wish a death upon bhyveload
and grub2-bhyve.

Thanks,

-- 
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:+1 443-546-8752
Tor+XMPP+OTR:latt...@is.a.hacker.sx
GPG Key ID:  0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE


signature.asc
Description: PGP signature

Re: bhyve: Detecting that a guest kernel has booted

[Rodney W. Grimes]

> On 2019-03-11T13:27:23 -0400
> Shawn Webb  wrote:
> >
> > Both FreeBSD and Linux supports virtio_console(4). I have no idea
> > about OpenBSD, but I'm sure they'd be open to an implementation if
> > asked.
> 
> Right.
> 
> > 
> > The NFS solution would work, but it would be somewhat fragile. What
> > happens when a VM crashes? What happens when the host crashes?
> 
> At least in my case:
> 
> If the VM crashes, it'll be restarted by a process supervisor (runit,
> here).

I think ping would suffice to make that determination?

> If the host crashes, I likely have bigger problems. In any case, I
> think that's still fine because all the host would care about is if the
> guest's file was touched more recently than the last time the host
> tried to start a bhyve process for it.

You could inside the vm simply touch /tmp/foo from cron and from the host
see that the diskimage last modified time updated.  Assuming some cacheing
does not get in the way.

> I suppose I should elaborate a bit: I do have monitoring via Prometheus
> in place, but I'd like to try to stagger VM startups a little as
> starting up a lot of them in parallel on boot tends to overwhelm the
> machine slightly. Once they're all up and running in a steady state,
> things are fine. I would typically stagger the startup of
> ordinary services (inside a jail, for example) by using dependencies in
> runit - it has a facility to pause a service until a dependent service
> has been started. Unfortunately, that can't work in this case because
> once the bhyve process has been started, the host can't tell if/when
> the guest has actually fully started up. As far as runit is concerned,
> the service is up and so any dependent services should be started too.

The package vmbhyve has starggered startup in a specific ordered list
implemented.  It is all writtin in /bin/sh, so easy to adapt.

> Mark Raynsford | http://www.io7m.com
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Rodney W. Grimes]

-- Start of PGP signed section.
> On 2019-03-11T13:08:53 -0400
> Shawn Webb  wrote:
> >
> > If your guest OS supports it, you could probably write two scripts that
> > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > received.
> > 
> 
> They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> one out of three of those supports it...
> 
> I suppose my other option would be to add (another) NFS mount in each
> guest, and have them touch a file early in the init script (and
> possibly touch a different file early in the shutdown script).

Well ICMP is in the kernel, and should be working as soon as the
interface is up, long before you could do anything with NFS,
so rather than the complexity above a simple ping would suffice.

There is also the phase of vmm(8) startup that when you are
running bhyveload vs bhyve and iirc grubload vs bhyve, that
can be detected.  vmbhyve does so and says you are in state
looader when you do a vm list.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Mark Raynsford via freebsd-virtualization]

On 2019-03-11T13:27:23 -0400
Shawn Webb  wrote:
>
> Both FreeBSD and Linux supports virtio_console(4). I have no idea
> about OpenBSD, but I'm sure they'd be open to an implementation if
> asked.

Right.

> 
> The NFS solution would work, but it would be somewhat fragile. What
> happens when a VM crashes? What happens when the host crashes?

At least in my case:

If the VM crashes, it'll be restarted by a process supervisor (runit,
here).

If the host crashes, I likely have bigger problems. In any case, I
think that's still fine because all the host would care about is if the
guest's file was touched more recently than the last time the host
tried to start a bhyve process for it.

I suppose I should elaborate a bit: I do have monitoring via Prometheus
in place, but I'd like to try to stagger VM startups a little as
starting up a lot of them in parallel on boot tends to overwhelm the
machine slightly. Once they're all up and running in a steady state,
things are fine. I would typically stagger the startup of
ordinary services (inside a jail, for example) by using dependencies in
runit - it has a facility to pause a service until a dependent service
has been started. Unfortunately, that can't work in this case because
once the bhyve process has been started, the host can't tell if/when
the guest has actually fully started up. As far as runit is concerned,
the service is up and so any dependent services should be started too.

--
Mark Raynsford | http://www.io7m.com



pgpJ0XYUfieQs.pgp
Description: OpenPGP digital signature

Re: bhyve: Detecting that a guest kernel has booted

[Shawn Webb]

On Mon, Mar 11, 2019 at 05:21:58PM +, Mark Raynsford wrote:
> On 2019-03-11T13:08:53 -0400
> Shawn Webb  wrote:
> >
> > If your guest OS supports it, you could probably write two scripts that
> > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > received.
> > 
> 
> They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> one out of three of those supports it...
> 
> I suppose my other option would be to add (another) NFS mount in each
> guest, and have them touch a file early in the init script (and
> possibly touch a different file early in the shutdown script).

Both FreeBSD and Linux supports virtio_console(4). I have no idea
about OpenBSD, but I'm sure they'd be open to an implementation if
asked.

The NFS solution would work, but it would be somewhat fragile. What
happens when a VM crashes? What happens when the host crashes?

Thanks,

-- 
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:+1 443-546-8752
Tor+XMPP+OTR:latt...@is.a.hacker.sx
GPG Key ID:  0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE


signature.asc
Description: PGP signature

Re: bhyve: Detecting that a guest kernel has booted

[Mark Raynsford via freebsd-virtualization]

On 2019-03-11T13:08:53 -0400
Shawn Webb  wrote:
>
> If your guest OS supports it, you could probably write two scripts that
> uses virtio_console(4), one for the guest to tell the host "HELLO" and
> one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> received.
> 

They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
one out of three of those supports it...

I suppose my other option would be to add (another) NFS mount in each
guest, and have them touch a file early in the init script (and
possibly touch a different file early in the shutdown script).

-- 
Mark Raynsford | http://www.io7m.com



pgpJiTf4XQzv0.pgp
Description: OpenPGP digital signature

Re: bhyve: Detecting that a guest kernel has booted

[Jason Barbier]

None I know off the top of my head nor would that be really something easily 
done without some sort of agent. We could probably make some sort of inference 
based calls to the emulated registers but even then that wouldn't be that 
correct.

---
Jason Barbier | E: ja...@corrupted.io 

On Mon, Mar 11, 2019, at 10:05 AM, Mark Raynsford via freebsd-virtualization 
wrote:
> Hello!
> 
> Leaving aside userland monitoring tools such as Prometheus[0], is there
> any way to detect on the host that a guest kernel running in bhyve has
> booted? I'm assuming "booted" in this sense to mean "PID 1 has probably
> started". I'm guessing that there probably isn't, but I thought I'd
> better ask anyway. "Not booted" would mean something like "Is sitting
> at the Grub prompt doing nothing".
> 
> [0] https://prometheus.io/
> 
> -- 
> Mark Raynsford | http://www.io7m.com
> 
> 
> Attachments:
> * null
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Shawn Webb]

On Mon, Mar 11, 2019 at 05:04:20PM +, Mark Raynsford via 
freebsd-virtualization wrote:
> Hello!
> 
> Leaving aside userland monitoring tools such as Prometheus[0], is there
> any way to detect on the host that a guest kernel running in bhyve has
> booted? I'm assuming "booted" in this sense to mean "PID 1 has probably
> started". I'm guessing that there probably isn't, but I thought I'd
> better ask anyway. "Not booted" would mean something like "Is sitting
> at the Grub prompt doing nothing".

If your guest OS supports it, you could probably write two scripts that
uses virtio_console(4), one for the guest to tell the host "HELLO" and
one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
received.

Of course, replace the "HELLO" and "NICE TO SEE YOU!" with the right
logic you're looking for. ;)

The "HELLO" bit could be written as an init script.

Thanks,

-- 
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:+1 443-546-8752
Tor+XMPP+OTR:latt...@is.a.hacker.sx
GPG Key ID:  0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE


signature.asc
Description: PGP signature

bhyve: Detecting that a guest kernel has booted

[Mark Raynsford via freebsd-virtualization]

Hello!

Leaving aside userland monitoring tools such as Prometheus[0], is there
any way to detect on the host that a guest kernel running in bhyve has
booted? I'm assuming "booted" in this sense to mean "PID 1 has probably
started". I'm guessing that there probably isn't, but I thought I'd
better ask anyway. "Not booted" would mean something like "Is sitting
at the Grub prompt doing nothing".

[0] https://prometheus.io/

-- 
Mark Raynsford | http://www.io7m.com



pgpMokSzb7UzS.pgp
Description: OpenPGP digital signature