Re: increasing bhyve VM_MAXCPU

[Rodney W. Grimes]

-- Start of PGP signed section.
> On Tue, May 18, 2021 at 02:51:22PM -0600, John Doherty via 
> freebsd-virtualization wrote:
> > I'm experimenting with bhyve VMs with more than 16 vCPUs on FreeBSD
> > 12.2-RELEASE.
> > 
> > As I understand it, this limit is defined as VM_MAXCPU in
> > /usr/src/sys/amd64/include/vmm.h. I also understand that as of about two
> > years ago, it should be possible to increase it beyond 21[1].
> > 
> > I began working on a machine with only eight CPU cores/threads, so rather
> > than trying to increase VM_MAXCPU, I tried to decrease it, which seemed to
> > work fine. My procedure was:
> > 
> >   edit /usr/src/sys/amd64/include/vmm.h
> >   cd /usr/src/lib/libvmmapi ; make clean ; make ; make install
> >   cd /usr/src/usr.sbin/bhyve ; make clean ; make ; make install
> >   cd /usr/src/usr.sbin/bhyvectl ; make clean ; make ; make install
> >   cd /usr/src/usr.sbin/bhyveload ; make clean ; make ; make install
> >   cd /usr/src ; make buildkernel KERNCONF=JLD ; make installkernel
> > KERNCONF=JLD
> >   reboot
> > 
> > where JLD is a kernel config file that differs from GENERIC only in the
> > "ident" line.
> > 
> > I tried several values (1, 2, and 4) for VM_MAXCPU and verified that VMs
> > with a number of vCPUs less than or equal to the max booted fine and that
> > VMs with more didn't.
> > 
> > So then I tried increasing VM_MAXCPU on a machine with two CPUs, 14 cores/28
> > threads each, using the same procedure. I set VM_MAXCPU to 28 and configured
> > a VM with 24 vCPUs but that didn't boot. Also tried setting the max to 20
> > and booting a VM with 18 vCPUs but that didn't work either. In all cases,
> > VMs with 16 or fewer vCPUs continued to boot fine.
> > 
> > All the tests were done with VMs created before doing any of this stuff. I
> > am using vm-bhyve to manage VMs. The VMs in these tests were running one of
> > FreeBSD 12.2-RELEASE, alpine linux 3.13, and debian-testing.
> > 
> > So it seems like I'm overlooking something but I don't know what, or maybe
> > this just isn't going to work. Anybody have experience actually doing this?
> > 
> > Thanks very much for any advice or further information.
> 
> HardenedBSD bumped max vCPUs up to 48 in a recent commit[0]. Our
> 14-CURRENT/amd64 package building VM is set up with 30 (out of 36
> available) vCPUs (`vmrun.sh -c 30`). We've completed one full package
> build (so, 31,000+ ports) without issue. The one package building VM
> we have on the system (so no other VMs) is running HardenedBSD
> 14-CURRENT. Meaning, we don't run anything other than HardenedBSD VMs.
> 
> [0]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/issues/30

When I made the changes to fix the brokeness at 21 vCPU we (Michael Dexter
and myself) did testing into the 200's of vcpu's.  Note that many guests
have problems at such large core counts, but bhyve itself behaved ok, but
some what sluggish as iirc the testing was done on a 32 core system.

> Thanks,
> 
> -- 
> Shawn Webb
> Cofounder / Security Engineer
> HardenedBSD
> 
> https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
-- End of PGP section, PGP failed!

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: increasing bhyve VM_MAXCPU

[Rodney W. Grimes]

> I'm experimenting with bhyve VMs with more than 16 vCPUs on FreeBSD 
> 12.2-RELEASE.
> 
> As I understand it, this limit is defined as VM_MAXCPU in 
> /usr/src/sys/amd64/include/vmm.h. I also understand that as of about two 
> years ago, it should be possible to increase it beyond 21[1].
> 
> I began working on a machine with only eight CPU cores/threads, so 
> rather than trying to increase VM_MAXCPU, I tried to decrease it, which 
> seemed to work fine. My procedure was:
> 
>edit /usr/src/sys/amd64/include/vmm.h

cp /usr/src/sys/amd64/include/vmm.h /usr/include/machine/vmm.h

>cd /usr/src/lib/libvmmapi ; make clean ; make ; make install
>cd /usr/src/usr.sbin/bhyve ; make clean ; make ; make install
>cd /usr/src/usr.sbin/bhyvectl ; make clean ; make ; make install
>cd /usr/src/usr.sbin/bhyveload ; make clean ; make ; make install

All of this code you built used the old value as you did not
propogate your change to vmm.h to the installed system, since
your running individual make commands the includes in /usr/include
are used.

>cd /usr/src ; make buildkernel KERNCONF=JLD ; make installkernel 
> KERNCONF=JLD
>reboot

Your kernel has the new value you put in vmm.h, so now you
have created a userland/kernel missmatch on some data structures.

> 
> where JLD is a kernel config file that differs from GENERIC only in the 
> "ident" line.
> 
> I tried several values (1, 2, and 4) for VM_MAXCPU and verified that VMs 
> with a number of vCPUs less than or equal to the max booted fine and 
> that VMs with more didn't.
> 
> So then I tried increasing VM_MAXCPU on a machine with two CPUs, 14 
> cores/28 threads each, using the same procedure. I set VM_MAXCPU to 28 
> and configured a VM with 24 vCPUs but that didn't boot. Also tried 
> setting the max to 20 and booting a VM with 18 vCPUs but that didn't 
> work either. In all cases, VMs with 16 or fewer vCPUs continued to boot 
> fine.
> 
> All the tests were done with VMs created before doing any of this stuff. 
> I am using vm-bhyve to manage VMs. The VMs in these tests were running 
> one of FreeBSD 12.2-RELEASE, alpine linux 3.13, and debian-testing.
> 
> So it seems like I'm overlooking something but I don't know what, or 
> maybe this just isn't going to work. Anybody have experience actually 
> doing this?
> 
> Thanks very much for any advice or further information.
> 
> [1] 
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: resizing aarch64 image on arm64

[Rodney W. Grimes]

> Rodney W. Grimes wrote this message on Tue, Apr 06, 2021 at 06:01 -0700:
> > > tech-lists wrote this message on Mon, Apr 05, 2021 at 16:59 +:
> > > > Can anyone tell me how to resize
> > > > https://download.freebsd.org/ftp/releases/VM-IMAGES/13.0-RC5/aarch64/Latest/FreeBSD-13.0-RC5-arm64-aarch64.raw
> > > > 
> > > > on an amd64 host?
> > > 
> > > note that this has been unxz'd already.
> > > 
> > > truncate -s 6g FreeBSD-13.0-RC5-arm64-aarch64.raw # enlarge
> > > mdconfig -f FreeBSD-13.0-RC5-arm64-aarch64.raw# make device
> > > gpart recover md1 # fix up gpt
> > > gpart resize -i 3 md1 # expand ufs 
> > > partition
> > > growfs md1p3  # grow ufs
> > > 
> > > Note that mdconfig may output a different md device than md1, use
> > > that instead.
> > 
> > NICE!  Maybe this could be a second example in growfs(8) as
> > an example for "How to prepare an image for use with emulation"?
> 
> In most/all cases for images, just a truncate is needed as Glen
> mentioned.  The growfs(7) rc script is run at first boot to size
> up the root fs as needed.  It can always be run manually later via:
> service growfs onestart

Though that is true, I suspect that running the growfs on
the host is much faster than doing so inside of say.. QEMU,
and is probably 10x faster than doing it before it ends up
on a SD card, though it would increase the dd time to write
the now much larger image.

Also one may be pre-growing the file system to add a bunch of
files to it before booting, I personally shall be using this
technique to build sd cards for arm boards now that I know
how to do it.

> 
> if/when the size of the disk changes...
> 
> Hmm, just noticed that growfs(7) isn't linked from growfs(8), guess
> I need to fix that.
> 
> -- 
>   John-Mark GurneyVoice: +1 415 225 5579
> 
>  "All that I will do, has been done, All that I have, has not."
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: resizing aarch64 image on arm64

[Rodney W. Grimes]

> tech-lists wrote this message on Mon, Apr 05, 2021 at 16:59 +:
> > Can anyone tell me how to resize
> > https://download.freebsd.org/ftp/releases/VM-IMAGES/13.0-RC5/aarch64/Latest/FreeBSD-13.0-RC5-arm64-aarch64.raw
> > 
> > on an amd64 host?
> 
> note that this has been unxz'd already.
> 
> truncate -s 6g FreeBSD-13.0-RC5-arm64-aarch64.raw # enlarge
> mdconfig -f FreeBSD-13.0-RC5-arm64-aarch64.raw# make device
> gpart recover md1 # fix up gpt
> gpart resize -i 3 md1 # expand ufs partition
> growfs md1p3  # grow ufs
> 
> Note that mdconfig may output a different md device than md1, use
> that instead.

NICE!  Maybe this could be a second example in growfs(8) as
an example for "How to prepare an image for use with emulation"?


>   John-Mark GurneyVoice: +1 415 225 5579
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Any plans to increase the maximum number of virtual CPUs above 16?

[Rodney W. Grimes]

-- Start of PGP signed section.
> On Tue, Nov 27, 2018 at 10:06:52PM +, Simon Connah wrote:
> > I was reading the bhyve man page and noticed that it said the maximum number
> > of virtual CPUs is limited to 16. With AMD Epyc being out with 32 cores and
> > 64 threads per CPU and being able to have 2 of them in one system this
> > number seems rather low by modern standards.
> > 
> > Are there any plans to increase this number to something like 32 virtual
> > CPUs?
> 
> I'm running into this. I have a server that can handle way more than
> 16 vCPUs. I know this topic has been talked about several times, and
> theoretical discussions keep taking place.
> 
> A bit more than two years later, and this is still a topic of
> discussion. I'm curious if anyone has a patch to punish me with. I'm
> very happy to be a guinea pig.

Several in phab, probably need rebased, its my WIP to eliminate
the compile time constant.  If you need more vCPU's it is just
a recompile of the kernel, libvmm, and bhyve(8) iirc.  Up the
value of MAX_VCPU.

https://reviews.freebsd.org/D10070
https://reviews.freebsd.org/D18816
https://reviews.freebsd.org/D18846

I believe all the fixes that allow you to increase VM_MAXCPU
up to something like 240 have all been committed.  

> Thanks,
> 
> -- 
> Shawn Webb
> Cofounder / Security Engineer
> HardenedBSD
> 
> GPG Key ID:  0xFF2E67A277F8E1FA
> GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9  3633 C85B 0AF8 AB23 0FB2
> https://git-01.md.hardenedbsd.org/HardenedBSD/pubkeys/src/branch/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
-- End of PGP section, PGP failed!

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [bhyve][arm64] Number of allowed vcpus

[Rodney W. Grimes]

> On Wed, Jan 13, 2021 at 4:55 PM Rodney W. Grimes <
> freebsd-...@gndrsh.dnsmgr.net> wrote:
> 
> > > Hello,
> > >
> > > >From my perspective bhyve should use what is stated in the device tree
> > if
> > > no parameter is given and throw an error when N is greater than the
> > device
> > > tree value.
> > >
> > > Mihai
> >
> > That seems to be one reasonable solution.  I am not very informed on what
> > the CPU over commit situation is on an ARM cpu with respect to
> > virtualization.
> >
> > I would also propose as an alternative to just do what was asked by the
> > user, which is the current amd64 implementation, OR do as asked and
> > emmit a warning message.
> >
> > I would ask given what was said above: isnt this the same situation as
> > if the sum of all VM VCPU's is more than the total cores in the system?
> > That is a very common situation, and one we must not restrict.
> >
> 
> We are talking here about the device tree of the guest. If you boot up a
> guest with a device tree of X vcpus, why specify something else on the
> command line?

Oh, that is/was not clear from the original post, it did not specify
which device tree was being discussed.  How do we even see the guest
device tree (again, pointing to my lack of knowledge on the ARM bhyve
implementation.)

Also would this not be about the same as taking my flash card out of
a 4 CPU system and plugging it into a 8 CPU system without updating
the device tree?  What would real hardware do here?

> > > On Wed, Jan 13, 2021 at 1:55 PM Andrei Martin <
> > andrei.cos.mar...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > While working on enabling multiple virtual cpus for the arm64 virtual
> > > > machine
> > > > I used the bhyve's "-c N" parameter to create N cpus (it gets only the
> > > > cpus, in contrast to the amd64). If N is smaller than the number of
> > cores
> > > > described in the device tree it will enable only N and if it's larger
> > it
> > > > will enable only the number of cores from the device tree.
> > > >
> > > > I don't know if it is the best approach. Another way would be to enable
> > > > all the cpus from the device tree, no more, no less.
> > > >
> > > > Do you have any suggestions?
> > > >
> > > > Andrei
> > > >
> > > ___
> > > freebsd-virtualization@freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > > To unsubscribe, send any mail to "
> > freebsd-virtualization-unsubscr...@freebsd.org"
> > >
> >
> > --
> > Rod Grimes
> > rgri...@freebsd.org
> >

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [bhyve][arm64] Number of allowed vcpus

[Rodney W. Grimes]

> Hello,
> 
> >From my perspective bhyve should use what is stated in the device tree if
> no parameter is given and throw an error when N is greater than the device
> tree value.
> 
> Mihai

That seems to be one reasonable solution.  I am not very informed on what
the CPU over commit situation is on an ARM cpu with respect to virtualization.

I would also propose as an alternative to just do what was asked by the
user, which is the current amd64 implementation, OR do as asked and
emmit a warning message.

I would ask given what was said above: isnt this the same situation as
if the sum of all VM VCPU's is more than the total cores in the system?
That is a very common situation, and one we must not restrict.

> On Wed, Jan 13, 2021 at 1:55 PM Andrei Martin 
> wrote:
> 
> > Hello,
> >
> > While working on enabling multiple virtual cpus for the arm64 virtual
> > machine
> > I used the bhyve's "-c N" parameter to create N cpus (it gets only the
> > cpus, in contrast to the amd64). If N is smaller than the number of cores
> > described in the device tree it will enable only N and if it's larger it
> > will enable only the number of cores from the device tree.
> >
> > I don't know if it is the best approach. Another way would be to enable
> > all the cpus from the device tree, no more, no less.
> >
> > Do you have any suggestions?
> >
> > Andrei
> >
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve with wlan0

[Rodney W. Grimes]

> As far as I remember it works just fine other than the standard caveats about 
> bridging over wireless interfaces which involve some odd inconsistencies that 
> could crop up from time to time especially with like DHCP, but a lot of that 
> depends on your network config as a whole. 
> The approach I would consider taking for a wireless host is if the guest 
> doesn't need direct access to the network just do a NAT network or a routed 
> network and route through the wlan interface instead of trying to do bridging 
> with it directly.

One very problematic area for wireless and bhyve is that each guests have a 
different MAC, wireless does NOT like that, as the AP usually associates with 
one and only one MAC address.

> ---
> Jason Barbier | E: ja...@corrupted.io 
> GPG: FD7D2D5F0A0FBE39 (https://keybase.io/kusuriya)
> 
> On Fri, Nov 27, 2020, at 09:03, tech-lists wrote:
> > Hi,
> > 
> > As subject: can a machine that only uses wlan0 for its connectivity
> > be a bhyve host? Context is 12.2-stable r367858 amd64.
> > 
> > Asking because ISTR some issue with bridging, but that was a while ago.
> > I'm anticipating managing instances with vm-bhyve. Previously have set
> > instances manually on other server hardware.
> > 
> > thanks,
> > -- 
> > J.
> > 
> > Attachments:
> > * signature.asc
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: nbdkit on FreeBSD

[Rodney W. Grimes]

> On Wed, Jun 17, 2020 at 9:53 PM Rodney W. Grimes <
> freebsd-...@gndrsh.dnsmgr.net> wrote:
> 
> > > Does anybody use nbdkit on FreeBSD?  It's a fancy NBD (Network Block
> > > Device) server.  It runs fine on FreeBSD, but there's no port.  If
> > anybody
> > > is interested, I'll make a port for it.  However, if I'm the only one
> > then
> > > I won't bother.
> > >
> > > https://github.com/libguestfs/nbdkit/
> >
> > I am not sure if this is what I looked at, but I was looking
> > for a way to do old style sun "nd", and this looks to be 1/2
> > of that, in that I see a server, but no client for the kernel.
> >
> > I would be interested in NBD if we have both ends.
> >
> 
> There's lots of 'other ends' including qemu for this.

I specifically meant native FreeBSD kernel.

> This is really cool!
> 
> Warner

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Cannot run FreeBSD 11 as a vm-bhyve guest under FreeBSD 12

[Rodney W. Grimes]

[ Charset UTF-8 unsupported, converting... ]
> On 3/3/20 11:18 AM, Andrea Venturoli wrote:
> > On 2020-03-01 16:58, Rodney W. Grimes wrote:
> >> One thing to try is to stop in the loader and do this:
> >> boot_serial=NO
> >> boot
> >
> > I "escaped to the loader prompt", but typing boot_serial=NO, yields 
> > "boot_serial=NO" not found. 
> 
> See the manual page for loader(8). The correct syntax is
> 
> set variable=value

Yes, my mistake, I grabbed the line out of my /boot/loader.conf file.

> Cheers,
> Mateusz Piotrowski
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Cannot run FreeBSD 11 as a vm-bhyve guest under FreeBSD 12

[Rodney W. Grimes]

> Hello.
> 
> I'd like to run a FreeBSD 11.3 VM under FreeBSD 12.1.
> As I'm already using vm-bhyve to run a Windows guest, I think the best 
> (or possibly only) choice is to use bhyve for the FreeBSD guest too.
> 
> I've got the following conf file:
> 
> > guest="freebsd"
> > loader="bhyveload"
> > cpu=1
> > memory=512M
> > network0_type="virtio-net"
> > network0_switch="public"
> > disk0_type="virtio-blk"
> > disk0_name="disk0"
> > disk0_dev="sparse-zvol"
> 
> Running "vm install -f f11b FreeBSD-11.3-RELEASE-amd64-disc1.iso", will 
> boot the guest, but it will hang after a while with:
> 
> > /boot/kernel/kernel text=0x1564b08 data=0x145330+0x4cdf30 
> > syms=[0x8+0x16daf0+0x8+0x186a43]
> > Booting...
>  > |
> 
> "vm stop" will stop it (through an ACIP shutdown) after a while.
> 
> 
> 
> So I tried with UEFI; here's the config file:
> > guest="freebsd"
> > loader="uefi"   
> > 
> > cpu=1
> > memory=512M
> > network0_type="virtio-net"
> > network0_switch="public"
> > disk0_type="virtio-blk"
> > disk0_name="disk0"
> > disk0_dev="sparse-zvol"
> > disk1_type="ahci-cd"
> > disk1_dev="custom"
> > disk1_name="/zroot/vm/.iso/FreeBSD-11.3-RELEASE-amd64-disc1.iso"
> > graphics="yes"
> > xhci_mouse="yes"
> > graphics_listen="192.168.xxx.1"
> 
> Starting it with "vm start" and connecting through VNC, it goes through 
> the same stages as above, but ends up with a blank screen with a white 
> block cursor in the first column of the first row.
> 
> 
> 
> Any hint on how to get past this?

THis again looks very much like the "lost console" issue of the system
flipping to a serial console and your trying to access it via a
graphical console interface.

One thing to try is to stop in the loader and do this:
boot_serial=NO
boot

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve issues on Dell C6220 node

[Rodney W. Grimes]

> Hello,
> 
> I've recently got hold of some Dell C6220 systems (2 servers in a single 
> chassis) while I was hoping to use for bhyve
> The basic spec of a single server is as follows -
> 
> Xeon(R) CPU E5-2670 (VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID)
> 64GB DDR3
> LSI MegaRaid 9265-8i
> FreeBSD 12.1
> 
> I had a bit of trouble getting the disks configured and booting (**probably 
> unrelated but see below for the details).
> 
> However, my main problem at the moment is that booting either 11.3 or 12.1 in 
> a bhyve guest (using bhyveload) gets as far as starting to boot from the 
> install ISO, then just hangs
> 
> "
> Loading kernel...
> /boot/kernel/kernel text=0x168fdf1 data=0x1d0a68+0x768d80 
> syms=[0x8+0x178bc0+0x8+0x1969d5]
> Loading configured modules...
> can't find '/boot/entropy'
> /
> "
> 
> I get either a "|" or "/" character, then nothing else.

My experience has been when I see this it is a "wrong console" issue, ie the 
kernel
has decided to use something else for a console and your output is going there.

It might be your setup using a UEFI with a fb console up to the end of the 
loader,
then the kernel decides it is using a serial console.  Or vice versa.  

> 
> I've also tested a windows server 2016 guest, and while that will actually 
> install via UEFI, it is noticeably slow - over a minute to boot to the login 
> screen and everything crawls along
> 
> I'm at a loss at the moment. My test machine for years has been an old 
> i3-2100 system and that has always booted freebsd & windows guests fairly well
> 
> I'd be grateful if anyone has any ideas or insight, or whether I just need to 
> scrap the whole idea and either switch to a completely different hypervisor 
> (which I'd really rather not do as I know FreeBSD very well and was planning 
> on making use of ZFS for backups/migration/etc) or find some different systems
> 
> 
> **Regarding the disks, I didn't really want a raid controller and have ended 
> up creating a jbod (RAID0) volume for each disk, which appears as mfidX. This 
> seems to work fine, although at first I tried to create a single pool of 2 
> mirrored pairs (4 disks). This failed to boot due to zfs block i/o errors on 
> boot. In the end I had to create a separate boot mirror across 2 partitions, 
> the create a second pool for actual data storage.
> 
> Regards,
> Matt Churchyard
> 
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve device emulation initialization error

[Rodney W. Grimes]

-- Start of PGP signed section.
> Hi,
> 
> I'm trying to install an old 11.1-release like this:
> 
> vmrun.sh -c 4 -m 32768M -tap2 -d test-host.img -i -I
  ^ is this a typo?  Should be -t tap2

> FreeBSD-11.1-RELEASE-amd64-dvd1.iso test-host
> 
> I get the boot screen and countdown but it fails to boot like this:
> 
> /boot/kernel/kernel text=0x14972f8 data=0x1384c0+0x4c15e8
> syms=[0x8+0x15e8b0+0x8+0x178422]
> Booting...
> device emulation initialization error: No such file or directory
> 
> ...and returns me to the command prompt.
> 
> Just in case it was an outated bhyve on this 12-stable r354992 system, I did
> this as root, on r355929 /usr/src:
> 
> # cd /usr/src/usr.sbin/bhyve && make && make install
> 
> installed without error
> 
> tried the vmrun.sh command again, same result.
> 
> Is there a way around this?
> 
> thanks,
> 
> 
> -- 
> J.
-- End of PGP section, PGP failed!

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Need X-server to run in bhyve guest

[Rodney W. Grimes]

-- Start of PGP signed section.
> Jason Tubnor wrote:
> > On Thu, 5 Dec 2019 at 12:32, Rodney W. Grimes 
> > 
> > wrote:
> > >
> > > TO fix some of your issues, ie the fact that the kernel boot
> > > messages and the installer do not appear in the vnc display
> > > if you drop to the loader prompt during the boot process and
> > > set serial_boot="NO"
> > > then you get the kernel boot messages and the installer in
> > > the vnc output.
> > >
> > > Once you finished the install adding:
> > > boot_serial="NO"
> > > to
> > > /boot/loader.conf
> > >
> > > works around the fact the kernel boot messages are going to a serial
> > > console.
> > >
> > >
> > > This *may* effect x11 startup, I havent got to that part yet...
> > >
> > >
> > More coverage on the above as Rod describes can be found on the wiki (down
> > near the bottom):
> > 
> > https://wiki.freebsd.org/bhyve/UEFI
> 
> Jason and Rodney, thanks, it's useful to know, though for the present I
> was happy enough with the serial console installation.
> 
> Now I'd like to concentrate on the question how to start a graphical
> desktop environment in a bhyve guest.

Ok, found the issue, or atleast a work around... scfb is not probing correctly,
so create:
/usr/local/etc/X11/xorg.conf.d/driver-scfb.conf
With this in it:

Section "Device"
 Identifier"Card0"
 Driver"scfb"
EndSection

Your x11 should then work fine  I tested with xfce

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Need X-server to run in bhyve guest

[Rodney W. Grimes]

-- Start of PGP signed section.
> Jason Tubnor wrote:
> > On Thu, 5 Dec 2019 at 12:32, Rodney W. Grimes 
> > 
> > wrote:
> > >
> > > TO fix some of your issues, ie the fact that the kernel boot
> > > messages and the installer do not appear in the vnc display
> > > if you drop to the loader prompt during the boot process and
> > > set serial_boot="NO"
> > > then you get the kernel boot messages and the installer in
> > > the vnc output.
> > >
> > > Once you finished the install adding:
> > > boot_serial="NO"
> > > to
> > > /boot/loader.conf
> > >
> > > works around the fact the kernel boot messages are going to a serial
> > > console.
> > >
> > >
> > > This *may* effect x11 startup, I havent got to that part yet...
> > >
> > >
> > More coverage on the above as Rod describes can be found on the wiki (down
> > near the bottom):
> > 
> > https://wiki.freebsd.org/bhyve/UEFI

Glad to see that this is documented.  I am unclear on why we are forcing
a serial console now though, perhaps it is a failure to detect the
prescense of a uefi framebuffer?

> 
> Jason and Rodney, thanks, it's useful to know, though for the present I
> was happy enough with the serial console installation.
> 
> Now I'd like to concentrate on the question how to start a graphical
> desktop environment in a bhyve guest.

I am back home now and souuld be able to dig out the vm I last did
this in, or recreate the situation and figure out what needs to be
done.

> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Need X-server to run in bhyve guest

[Rodney W. Grimes]

> Victor Sudakov wrote:
> > 
> > Fine, I will experiment with that. Maybe Gnome or xfce will recognize
> > this console as a monitor.
> > 
> > Thank you for the hint, Rodney!
> 
> Nope, would not run for me in the VNC console (see below). Did you do
> anything special?
> 
> /usr/local/bin/startxfce4: Starting X server
> 
> 
> X.Org X Server 1.18.4
> Release Date: 2016-07-19
> X Protocol Version 11, Revision 0
> Build Operating System: FreeBSD 12.0-RELEASE-p12 amd64 
> Current Operating System: FreeBSD uefi1.sibptus.ru 12.1-RELEASE FreeBSD 
> 12.1-RELEASE r354233 GENERIC amd64
> Build Date: 14 November 2019  08:46:03AM
>  
> Current version of pixman: 0.38.4
>   Before reporting problems, check http://wiki.x.org
>   to make sure that you have the latest version.
> Markers: (--) probed, (**) from config file, (==) default setting,
>   (++) from command line, (!!) notice, (II) informational,
>   (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
> (==) Log file: "/var/log/Xorg.0.log", Time: Mon Dec  2 23:28:28 2019
> (==) Using default built-in configuration (30 lines)
> scfb trace: probe start
> scfb trace: probe done
> (EE) 
> Fatal server error:
> (EE) Cannot run in framebuffer mode. Please specify busIDsfor all 
> framebuffer devices
> (EE) 
> (EE) 
> Please consult the The X.Org Foundation support 
>at http://wiki.x.org
>  for help. 
> (EE) Please also check the log file at "/var/log/Xorg.0.log" for additional 
> information.
> (EE) 
> (EE) Server terminated with error (1). Closing log file.
> xinit: giving up
> xinit: unable to connect to X server: Connection refused
> xinit: server error
> 

TO fix some of your issues, ie the fact that the kernel boot
messages and the installer do not appear in the vnc display
if you drop to the loader prompt during the boot process and
set serial_boot="NO"
then you get the kernel boot messages and the installer in
the vnc output.

Once you finished the install adding:
boot_serial="NO"
to
/boot/loader.conf

works around the fact the kernel boot messages are going to a serial console.


This *may* effect x11 startup, I havent got to that part yet...

> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Need X-server to run in bhyve guest

[Rodney W. Grimes]

> Rodney W. Grimes wrote:
> > > > 
> > > > > 
> > > > > Second, unless I select ZFS, there is no option to install on 
> > > > > GPT+UEFI.
> > > > > Fine, let's install on ZFS.
> > > > 
> > > > Do you mean GPT+UFS?  
> > > 
> > > I mean UFS+GPT+UEFI. That means, a EFI partition should be created and
> > > populated by the installer, alongside with the UFS and swap partitions.
> > > 
> > > bsdinstall can create and populate the EFI partition when
> > > installing on ZFS, but not when installing on UFS.
> > > 
> > > > Not sure what the failure is here.  I usually
> > > > install MBR/(UFS or ZFS) so this is not a use case for me.
> > > 
> > > How do you create the EFI partition, manually?
> > 
> > I don't use bsdinstall at all, I install using make and a Makefile,
> > so, kinda yes, kinda no, not manual, but not bsdinstall either.
> 
> I see. Well, I personally like bsdinstall for simple cases.  It's a pity
> it cannot automagically create a EFI partition when installing on UFS.
> But its ZFS installation routine is neat, and "bsdinstall jail" too.

I am not sure you actuall need an EFI partition in that situation, but
you may want one, and that is a missing feature of bsdinstall.
`
> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Need X-server to run in bhyve guest

[Rodney W. Grimes]

-- Start of PGP signed section.
> Rodney W. Grimes wrote:
> 
> [dd]
> 
> > 
> > > 
> > > Second, unless I select ZFS, there is no option to install on GPT+UEFI.
> > > Fine, let's install on ZFS.
> > 
> > Do you mean GPT+UFS?  
> 
> I mean UFS+GPT+UEFI. That means, a EFI partition should be created and
> populated by the installer, alongside with the UFS and swap partitions.
> 
> bsdinstall can create and populate the EFI partition when
> installing on ZFS, but not when installing on UFS.
> 
> > Not sure what the failure is here.  I usually
> > install MBR/(UFS or ZFS) so this is not a use case for me.
> 
> How do you create the EFI partition, manually?

I don't use bsdinstall at all, I install using make and a Makefile,
so, kinda yes, kinda no, not manual, but not bsdinstall either.

> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Need X-server to run in bhyve guest

[Rodney W. Grimes]

> On Tue, Dec 03, 2019 at 12:23:55AM -0800, Rodney W. Grimes wrote:
> > -- Start of PGP signed section.
> > > Victor Sudakov wrote:
> > > > 
> > > > Fine, I will experiment with that. Maybe Gnome or xfce will recognize
> > > > this console as a monitor.
> > > > 
> > > > Thank you for the hint, Rodney!
> > > 
> > > Nope, would not run for me in the VNC console (see below). Did you do
> > > anything special?
> > 
> > pkg add xf86-video-scfb
> > 
> 
> wow, please stop recommending pkg add, the end user should use pkg install! 
> pkg
> add is a compatiblity with old pkg_add to appease the ports tree and for some
> corner usage. All users should uses pkg install for a while now.

Your right,
finger memory, I still type it, get an error then retype pkg install :-)

> Best regards,
> Bapt

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Need X-server to run in bhyve guest

[Rodney W. Grimes]

-- Start of PGP signed section.
> Victor Sudakov wrote:
> > 
> > Fine, I will experiment with that. Maybe Gnome or xfce will recognize
> > this console as a monitor.
> > 
> > Thank you for the hint, Rodney!
> 
> Nope, would not run for me in the VNC console (see below). Did you do
> anything special?

pkg add xf86-video-scfb

> 
> /usr/local/bin/startxfce4: Starting X server
> 
> 
> X.Org X Server 1.18.4
> Release Date: 2016-07-19
> X Protocol Version 11, Revision 0
> Build Operating System: FreeBSD 12.0-RELEASE-p12 amd64 
> Current Operating System: FreeBSD uefi1.sibptus.ru 12.1-RELEASE FreeBSD 
> 12.1-RELEASE r354233 GENERIC amd64
> Build Date: 14 November 2019  08:46:03AM
>  
> Current version of pixman: 0.38.4
>   Before reporting problems, check http://wiki.x.org
>   to make sure that you have the latest version.
> Markers: (--) probed, (**) from config file, (==) default setting,
>   (++) from command line, (!!) notice, (II) informational,
>   (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
> (==) Log file: "/var/log/Xorg.0.log", Time: Mon Dec  2 23:28:28 2019
> (==) Using default built-in configuration (30 lines)
> scfb trace: probe start
> scfb trace: probe done

O.o it is installed

> (EE) 
> Fatal server error:
> (EE) Cannot run in framebuffer mode. Please specify busIDsfor all 
> framebuffer devices
> (EE) 
> (EE) 
> Please consult the The X.Org Foundation support 
>at http://wiki.x.org
>  for help. 
> (EE) Please also check the log file at "/var/log/Xorg.0.log" for additional 
> information.
> (EE) 
> (EE) Server terminated with error (1). Closing log file.
> xinit: giving up
> xinit: unable to connect to X server: Connection refused
> xinit: server error

I am traveling this week, I would need to look at VM's I can not
get to right now.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Need X-server to run in bhyve guest

[Rodney W. Grimes]

> Rodney W. Grimes wrote:
> > > Henrik Gulbrandsen wrote:
> > > > On 2019-12-01 06:35, Victor Sudakov wrote:
> > > > > If I were to test different desktop environments (gnome, xfce etc) in 
> > > > > a
> > > > > FreeBSD guest in bhyve, what X server (probably with vnc access, or 
> > > > > rdp
> > > > > access, or whatever?) would you advise?
> > > > 
> > > > This will work with the normal X server and bhyve's built-in VNC if
> > > > you run the guest with UEFI CSM and apply my Video BIOS patch:
> > > > 
> > > >  https://www.gulbra.net/freebsd-bhyve/
> > > 
> > > I'd prefer to stick to bhyveload for now.
> > 
> > I would strongly encourage you to use vnc and UEFI bios mode
> > in bhyve for what you are attempting to do.  Unless your guess
> > is not able to run a uefi frame buffer type display, which any
> > recent FreeBSD should be just fine with.
> 
> All right, I just tried to install FreeBSD 12.1 under vm-bhyve in UEFI
> mode. 
> 
> Well, the installation was not trivial.
> 
> At first, in the VNC client, I see the loader menu, then the "Loading
> kernel..." message, and then a blank screen. Fine, I run "vm console"
> and install via the serial console, the UEFI screen still being blank.

*sigh*  Thats broken  If I make it to this weeks bhyve call I'll bring
it up, this sounds like a loader to kernel handoff regression.

> 
> Second, unless I select ZFS, there is no option to install on GPT+UEFI.
> Fine, let's install on ZFS.

Do you mean GPT+UFS?  Not sure what the failure is here.  I usually
install MBR/(UFS or ZFS) so this is not a use case for me.

> Installation complete, reboot... The serial console is available, the
> UEFI VNC console is available too, though much later :-)

:-(

> Fine, I will experiment with that. Maybe Gnome or xfce will recognize
> this console as a monitor.

It should just be the scfb x11 driver.

> 
> Thank you for the hint, Rodney!

Glad to have moved you forward, even if it uncovered more sharp
pokey edges of bhyve in the process.

> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Need X-server to run in bhyve guest

[Rodney W. Grimes]

-- Start of PGP signed section.
> Henrik Gulbrandsen wrote:
> > On 2019-12-01 06:35, Victor Sudakov wrote:
> > > If I were to test different desktop environments (gnome, xfce etc) in a
> > > FreeBSD guest in bhyve, what X server (probably with vnc access, or rdp
> > > access, or whatever?) would you advise?
> > 
> > This will work with the normal X server and bhyve's built-in VNC if
> > you run the guest with UEFI CSM and apply my Video BIOS patch:
> > 
> >  https://www.gulbra.net/freebsd-bhyve/
> 
> I'd prefer to stick to bhyveload for now.

I would strongly encourage you to use vnc and UEFI bios mode
in bhyve for what you are attempting to do.  Unless your guess
is not able to run a uefi frame buffer type display, which any
recent FreeBSD should be just fine with.

You do not need the Video BIOS patch for a FreeBSD guest.

> > 
> > However, when I tested it with a GhostBSD 19.04 ISO image, the mouse
> > didn't work - probably because FreeBSD still didn't support the XHCI
> > tablet needed for absolute pointer coordinates in bhyve. With a PS/2
> > mouse you will get two mouse pointers - one each for host and guest.
> 
> -- 
> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
> 2:5005/49@fidonet http://vas.tomsk.ru/
-- End of PGP section, PGP failed!

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Inter-VM networking with FreeBSD under bhyve

[Rodney W. Grimes]

> Thanks for the replies, everyone.
> 
> > Do you have any desired parameters?
> > Performance Driven?  Simplicity?
> I failed to mention that.  I'm interested in performance, especially low
> CPU overhead.

Then a netmap based solution is probably in line with your goal.

> I had tried vale+tap on a stable/12 host, but it wouldn't pass traffic,
> so I started this thread as a feeler for what was expected to work on
> which branches.  Since then, I updated the host to head, and vale+tap works.
> 
> I'm very interested in vale+ptnetmap.  It seems to be partially
> committed to head, but not entirely.  Is that right?

I am not well informed as to the state of this.

> Thanks again.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Severely broken bhyve console

[Rodney W. Grimes]

> On 2019-08-20T14:34:02 -0700
> "Rodney W. Grimes"  wrote:
> >
> > What it looks as if is happened is you have disconnected from
> > a session while it was in grub and had sent specific control
> > sequences, possibly even expecting scroll regions and such
> > set up in the terminal.  Now when you reconnect from a newly
> > initialzed terminal that has none of this setup things go
> > very wrong.
> 
> Interesting. I don't remember this happening, but it's not
> inconceivable that it happened and I didn't notice it.

Is it possible the VM guest dropped out to grub, or is
in a failed reboot state, or in a partial boot state?
If grub started up while you had no console attached
it may of output a bunch of control sequences that now
went into the bit bucket that are need to set the scroll
regions, reverse video, etc.

Is there a way to tell grub it is using a dumb terminal
and not a PC console, or vt100, or xterm.  Just a dumb
old tty (/etc/termcap calls this "dumb".)  Using that
as the console type may resolve these issues.

> What is kind of strange is that I appear to be partly in the grub
> prompt and partly in the Debian installer; I see output and responses
> from both interleaved.

I agree that is very odd, is it possible the same nmdm device
somehow has been attached to multip vm's?  I am not even sure
if that is possible.  Traditional tty's can be open by more
than one process so I think it might be possible to do that,
and this would lead to a mess as well.

> > One solution is to use the graphics console, that does not suffer
> > from these tty type issues, its a bit sluggish if you do not have
> > a good high speed network path to the bhyve host though.
> 
> It's a 1Gbps LAN, so I assume that's fast enough.
> 
> However: Is there some way I can reset the nmdm device?

The problem is not that you need to reset the nmdm device,
the problem is what ever is running on the inside the
guest talking to the console believes the device to be in
a different state than what your device is in, and there
is no convienient way to get your device in the expected
state.

In the situation of a shell prompt you can get the 2 states
alligned by issueing a terminal reset command of some form,
I have no idea how to do that at a grub prompt.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Severely broken bhyve console

[Rodney W. Grimes]

> Hello!
> 
> For reasons I'm unable to explain, my ability to access the nmdm
> consoles of bhyve guests has suddenly broken. It may have been caused
> by a recent 11.2 update - I keep this machine on 11.2-RELEASE plus
> freebsd-update patches but, as it's not a publically accessible machine,
> I rarely reboot for updates unless absolutely necessary.
> 
> I can use "sudo cu -l /dev/nmdm*B" to connect to a particular bhyve
> guest's console, but as this video shows, the console is effectively
> unusable:
> 
> https://ataxia.io7m.com/2019/08/20/broken_terminal.ogv
> 
> In that video, I connect to /dev/nmdm57B when the guest is shut down. I
> start the guest in a separate terminal (not pictured) and wait for it
> to boot into the Debian installer. I don't touch the keys until the
> title bar becomes visible. At that point, touching the arrow keys or
> the return key results in chaos.
> 
> It doesn't seem to be terminal-specific; I'm ssh'd into the host
> machine, and have tried various different terminals and $TERM settings
> to no effect. It doesn't seem to be guest-specific; I'm running
> FreeBSD, Debian, and OpenBSD guests and they now all show this
> behaviour. The guests themselves don't appear to be in any distress, I
> can ssh into the guests and everything is working correctly.
> 
> The host:
> 
> # uname -a
> FreeBSD cranberry.int.arc7.info 11.2-RELEASE-p13 FreeBSD
> 11.2-RELEASE-p13 #0: Tue Aug  6 06:41:33 UTC 2019
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
> 
> I'd appreciate any help on this.

I see this often, but normally I have a shell prompt and can clear
the problem with a
resizewin
command
or a
stty sane

Other things to try are clear, reset, tset.  But none of that is
going to work inside a grub> prompt.

What it looks as if is happened is you have disconnected from
a session while it was in grub and had sent specific control
sequences, possibly even expecting scroll regions and such
set up in the terminal.  Now when you reconnect from a newly
initialzed terminal that has none of this setup things go
very wrong.

This can happen on real hardware if you power down your vt320,
and then power it back up while something has done a lot of
"special" stuff to setup the screen.

One solution is to use the graphics console, that does not suffer
from these tty type issues, its a bit sluggish if you do not have
a good high speed network path to the bhyve host though.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Inter-VM networking with FreeBSD under bhyve

[Rodney W. Grimes]

> freebsd-virtualization:
> 
> What is my best option for inter-VM networking with FreeBSD guests under
> bhyve?  I'm only concerned with networking between VMs, not with the
> host or physical network.  I imagine some form of netmap is my best bet,
> but what exactly?
> 
> Thanks in advance,

Do you have any desired parameters?
Performance Driven?  Simplicity?  

netmap would probably be the most performant solution,
A host bridge device with taps per guest with a vtnet device
in the guest would probably be the simplist.

By not attaching the bridge to a host adapter and not giving it
an IP on the host the host is left out of the communications, BUT
it can debug most any of it with tcpdump.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: New video BIOS for bhyve

[Rodney W. Grimes]

> On 7/19/19 3:02 AM, Henrik Gulbrandsen wrote:
> > In the last few months, I have been extending the video support in bhyve
> > to allow booting from live ISO images, which sometimes lack UEFI 
> > loaders.
> > There was already a video BIOS with dormant support for VGA, but I could
> > not use that, because the X11 int10 module runs a real-mode x86 emulator
> > that aborts when it tries to enter the C code running in protected mode.
> > 
> > Instead, I added support for PCI Expansion ROMs to bhyve and wrote a new
> > SVGA video BIOS from scratch in 8086 assembly code. The current features
> > include VGA modes 3 and 12h, 24-bit and 32-bit VESA modes from 640x480 
> > to
> > 1920x1200, and enough BIOS functions to boot a typical Linux desktop.
> > 
> > At this point, I think it's good enough for a preliminary release, so if
> > you are interested, you will find all patch files at the following URL:
> > 
> >  https://www.gulbra.net/freebsd-bhyve/
> > 
> > My apologies for the 15-year-old web layout, which will probably break
> > horribly on modern smartphones. Also, I hope I have not duplicated the
> > work of someone else. I didn't want to announce this before it worked.
> 
> This is pretty impressive work!  Would you be able to post these patches to
> a public repository, e.g. on github or gitlab or some such?  That would
> really help with review and allow others to assist with the work as well.

I let this fall through my cracks as well while busy with
other things.  I too would like to see this work moved
forward.  I see that your patches are against an 11.3
system, and can appreciate that not everone has resources
to run -head, how can we assist in getting these patches
moved forward to head?

Regards,
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: ARMv8 development board with GICv3

[Rodney W. Grimes]

> If anyone finds any information on raspberry pi4 and it's bcm2711 that just
> released I will be very interested to see if it's a suitable testing
> platform. Not able to find documentation yet to validate that the interrupt
> controller. Rest looks good.

A little bit of foot work and things tell me that chip has
a GIC400 in it, and the GIC400 says:

GIC-400 can be configured to support only the required number
of cores and interrupts to reduce gate count. GIC-400
implements GICv2 architecture, Security and Virtualization
Extensions, for more information see Arm Generic Interrupt
Controller Architecture version 2.0.

So, no, this is not a V3 gic, so no bhyve/arm is not going to work
on it.

> On Sat, Jun 16, 2018 at 5:32 AM Marcin Wojtas  wrote:
> 
> > 2018-06-15 20:40 GMT+02:00 Alexandru Elisei :
> > >
> > > On Fri, Jun 15, 2018 at 8:51 PM, Jim Thompson  wrote:
> > > >
> > > >
> > > > https://gist.github.com/gonzopancho/760ab9ecee9dfbc1b6033e48647a4b48
> > > >
> > > > Note: we haven?t upstreamed everything.  Yet.
> > >
> > > Is FreeBSD bootable on the board and the NIC working?
> >
> > In 2017 I pushed patches and the board was booting with the network,
> > USB and SATA. However I know that loos from Netgate has improved the
> > overall support - as a result we can see the log Jim provided.
> >
> > >
> > > > macchiato.bin needs interrupts and NIC drivers, and seems a better
> > target
> > > > for bhyve
> > > >
> > > > machiatto.bin 4C A72 @ 1.6GHz/2.0GHz,  16GB DDR4
> > > >
> > > > espresso.bin 2C A53 @1.0GHz and1/2GB DDR3 (*)
> >
> > Just one note - macchiato.bin is using GICv2m and espresso - GICv3.
> >
> > Best regards,
> > Marcin
> >
> > > >
> > > > Jim
> > > >
> > > > (*) Very new espresso.bin has 1.2GHz SoC and DDR4, and you can plug a
> > SATA
> > > > drive in, including power.
> > > > https://i.imgur.com/5fc2vv0.jpg

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Windows 10 guests fail to boot when attempting to passthrough network card

[Rodney W. Grimes]

[ Charset UTF-8 unsupported, converting... ]
> On 5/21/19 9:48 AM, Rodney W. Grimes wrote:
> >> On 5/19/19 6:24 AM, Rodney W. Grimes wrote:
> >>>> Does the card have multiple BARs? Can you send us the output of pciconf 
> >>>> -lvb for the card?
> >>>>
> >>>> There is a bug in bhyve where if the OS assigns the 0x sentinel 
> >>>> to more than one BAR simultaneously it is not compatible with bhyve?s 
> >>>> memory model for the guest.
> >>>
> >>> The "we talked" below is myself and Nick talking about exactly that
> >>> which came from jhb@ pointing out to me in a hall wall
> >>> conversation here at BSDCan that this is the cause of this
> >>> problem and is a known issue.
> >>>
> >>> Does anyone have fixes for this issue?
> >>
> >> I have not tested this at all, but I have a candidate patch to fix the root
> >> issue at 
> >> https://github.com/freebsd/freebsd/compare/master...bsdjhb:bhyve_passthrough_barsize
> > 
> > Could you please factor out the white space changes and do that
> > seperately and sooner, it clouds the S/N in this review a bit.
> 
> I can pull those out, but this is just a testing patch still, not yet a commit
> candidate.

Thanks and even more reason to pull those out, IMHO

> -- 
> John Baldwin
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Windows 10 guests fail to boot when attempting to passthrough network card

[Rodney W. Grimes]

> On 5/19/19 6:24 AM, Rodney W. Grimes wrote:
> >> Does the card have multiple BARs? Can you send us the output of pciconf 
> >> -lvb for the card?
> >>
> >> There is a bug in bhyve where if the OS assigns the 0x sentinel to 
> >> more than one BAR simultaneously it is not compatible with bhyve?s memory 
> >> model for the guest.
> > 
> > The "we talked" below is myself and Nick talking about exactly that
> > which came from jhb@ pointing out to me in a hall wall
> > conversation here at BSDCan that this is the cause of this
> > problem and is a known issue.
> > 
> > Does anyone have fixes for this issue?
> 
> I have not tested this at all, but I have a candidate patch to fix the root
> issue at 
> https://github.com/freebsd/freebsd/compare/master...bsdjhb:bhyve_passthrough_barsize

Could you please factor out the white space changes and do that
seperately and sooner, it clouds the S/N in this review a bit.

> My first guess at the problem wasn't quite right (though Windows does indeed 
> size
> BARs in parallel).  For pass-through devices we aren't currently correctly 
> disabling
> access to a BAR when the guest disables it in the PCI command register which 
> in
> turn trips the assertion when using a Windows guest, but in general we need 
> to be
> honoring the enable bits when set or cleared by the guest.

That makes more since to me.  Thanks.

> -- 
> John Baldwin
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Windows 10 guests fail to boot when attempting to passthrough network card

[Rodney W. Grimes]

> Does the card have multiple BARs? Can you send us the output of pciconf -lvb 
> for the card?
> 
> There is a bug in bhyve where if the OS assigns the 0x sentinel to 
> more than one BAR simultaneously it is not compatible with bhyve?s memory 
> model for the guest.

The "we talked" below is myself and Nick talking about exactly that
which came from jhb@ pointing out to me in a hall wall
conversation here at BSDCan that this is the cause of this
problem and is a known issue.

Does anyone have fixes for this issue?

> 
> On Sat, May 18, 2019 at 21:14, Nick Principe  wrote:
> 
> > Yeah, that triggers a barf with error 2:
> > mmio_rb_lookup: barf ENOENTerror is 2
> > mr.name is passthru-pci-5
> > mr.flags is 0
> > mr.arg2 is 0
> > mr.base is 18446744073709027328
> > mr.size is 524288
> > Assertion failed: (error == 0), function modify_bar_registration, file
> > /usr/src/usr.sbin/bhyve/pci_emul.c, line 510.
> >
> > Interestingly enough, after we talked I simply commented out the assert at
> > 504 in pci_emul.c and Windows booted fine and can see the Chelsio card with
> > both interfaces. I'm not sure if it would fall over once I started actually
> > using it or not.
> >
> > `nap
> >
> > On Sat, May 18, 2019 at 4:06 PM Rodney W. Grimes <
> > freebsd-...@gndrsh.dnsmgr.net> wrote:
> >
> >> > I have noticed that Windows 10 guests fail to boot when attempting to
> >> pass
> >> > through a network card. I believe I have observed this with both cxgbe
> >> > (t580) and mlx5en cards, but only have a cxgbe to test with now. Without
> >> > passthrough, the Windows 10 guest boots and operates normally.
> >> >
> >> > FreeBSD guests (12.0-RELEASE) have no issue when booting with the cxgbe
> >> > card passed through - I can kldload cxgbe and I get both cxl ports
> >> showing
> >> > up in the FreeBSD guest.
> >> >
> >> > I have tested this with both 12.0-RELEASE and head (13-CURRENT r347883)
> >> as
> >> > the host OS with no change in behavior. The bhyve output is:
> >> > Unhandled ps2 keyboard command 0x02
> >> > Unhandled ps2 keyboard command 0x02
> >> > Assertion failed: (error == 0), function modify_bar_registration, file
> >> > /usr/src/usr.sbin/bhyve/pci_emul.c, line 504.
> >> > fbuf frame buffer base: 0x94360 [sz 16777216]
> >> >
> >> > Two main suggestions from discussions at BSDCan this week were:
> >> > - Capture pciconf -lvb from the FreeBSD guest
> >> > - Add some printf to pci_emul.c to capture some values when there is an
> >> > error
> >> >
> >> > I've captured the above, and a lot of other relevant info, in a Google
> >> Doc
> >> > here (too big to post directly):
> >> >
> >> https://docs.google.com/document/d/1t-UVIO9Aq0TPUFHyo1nVscqaW1LoPuNhfLPitL8oeTs/edit?usp=sharing
> >> >
> >> > `nap
> >>
> >> To confirm what I see from looking at your data could you tell me if this
> >> patch triggers a barf?
> >>
> >> --- mem.c.orig 2019-05-18 20:04:26.707995000 +
> >> +++ mem.c 2019-05-18 20:04:02.205119000 +
> >> @@ -97,6 +97,7 @@
> >> return (0);
> >> }
> >>
> >> +printf("mmio_rb_lookup: barf ENOENT");
> >> return (ENOENT);
> >> }
> >>
> >> --
> >> Rod Grimes
> >> rgri...@freebsd.org
> >>
> >
> > --
> > Nick Principe
> > Performance Engineering Supervisor
> > iXsystems, Inc.
> > Ph: (408) 943-4100 x341
> > Fx: (408) 943-4101
> > ___
> > freebsd-virtualization@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to 
> > "freebsd-virtualization-unsubscr...@freebsd.org"
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Windows 10 guests fail to boot when attempting to passthrough network card

[Rodney W. Grimes]

> Yeah, that triggers a barf with error 2:
> mmio_rb_lookup: barf ENOENTerror is 2
> mr.name is passthru-pci-5
> mr.flags is 0
> mr.arg2 is 0
> mr.base is 18446744073709027328
> mr.size is 524288
> Assertion failed: (error == 0), function modify_bar_registration, file
> /usr/src/usr.sbin/bhyve/pci_emul.c, line 510.
> 
> Interestingly enough, after we talked I simply commented out the assert at
> 504 in pci_emul.c and Windows booted fine and can see the Chelsio card with
> both interfaces. I'm not sure if it would fall over once I started actually
> using it or not.

I would go forward with testing, worst that happens is a host panic.


> `nap
> 
> 
> On Sat, May 18, 2019 at 4:06 PM Rodney W. Grimes <
> freebsd-...@gndrsh.dnsmgr.net> wrote:
> 
> > > I have noticed that Windows 10 guests fail to boot when attempting to
> > pass
> > > through a network card. I believe I have observed this with both cxgbe
> > > (t580) and mlx5en cards, but only have a cxgbe to test with now. Without
> > > passthrough, the Windows 10 guest boots and operates normally.
> > >
> > > FreeBSD guests (12.0-RELEASE) have no issue when booting with the cxgbe
> > > card passed through - I can kldload cxgbe and I get both cxl ports
> > showing
> > > up in the FreeBSD guest.
> > >
> > > I have tested this with both 12.0-RELEASE and head (13-CURRENT r347883)
> > as
> > > the host OS with no change in behavior. The bhyve output is:
> > > Unhandled ps2 keyboard command 0x02
> > > Unhandled ps2 keyboard command 0x02
> > > Assertion failed: (error == 0), function modify_bar_registration, file
> > > /usr/src/usr.sbin/bhyve/pci_emul.c, line 504.
> > > fbuf frame buffer base: 0x94360 [sz 16777216]
> > >
> > > Two main suggestions from discussions at BSDCan this week were:
> > > - Capture pciconf -lvb from the FreeBSD guest
> > > - Add some printf to pci_emul.c to capture some values when there is an
> > > error
> > >
> > > I've captured the above, and a lot of other relevant info, in a Google
> > Doc
> > > here (too big to post directly):
> > >
> > https://docs.google.com/document/d/1t-UVIO9Aq0TPUFHyo1nVscqaW1LoPuNhfLPitL8oeTs/edit?usp=sharing
> > >
> > > `nap
> >
> > To confirm what I see from looking at your data could you tell me if this
> > patch triggers a barf?
> >
> > --- mem.c.orig  2019-05-18 20:04:26.707995000 +
> > +++ mem.c   2019-05-18 20:04:02.205119000 +
> > @@ -97,6 +97,7 @@
> > return (0);
> > }
> >
> > +printf("mmio_rb_lookup: barf ENOENT");
> > return (ENOENT);
> >  }
> >
> > --
> > Rod Grimes
> > rgri...@freebsd.org
> >
> 
> 
> -- 
> Nick Principe
> Performance Engineering Supervisor
> iXsystems, Inc.
> Ph: (408) 943-4100 x341
> Fx: (408) 943-4101

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Windows 10 guests fail to boot when attempting to passthrough network card

[Rodney W. Grimes]

> I have noticed that Windows 10 guests fail to boot when attempting to pass
> through a network card. I believe I have observed this with both cxgbe
> (t580) and mlx5en cards, but only have a cxgbe to test with now. Without
> passthrough, the Windows 10 guest boots and operates normally.
> 
> FreeBSD guests (12.0-RELEASE) have no issue when booting with the cxgbe
> card passed through - I can kldload cxgbe and I get both cxl ports showing
> up in the FreeBSD guest.
> 
> I have tested this with both 12.0-RELEASE and head (13-CURRENT r347883) as
> the host OS with no change in behavior. The bhyve output is:
> Unhandled ps2 keyboard command 0x02
> Unhandled ps2 keyboard command 0x02
> Assertion failed: (error == 0), function modify_bar_registration, file
> /usr/src/usr.sbin/bhyve/pci_emul.c, line 504.
> fbuf frame buffer base: 0x94360 [sz 16777216]
> 
> Two main suggestions from discussions at BSDCan this week were:
> - Capture pciconf -lvb from the FreeBSD guest
> - Add some printf to pci_emul.c to capture some values when there is an
> error
> 
> I've captured the above, and a lot of other relevant info, in a Google Doc
> here (too big to post directly):
> https://docs.google.com/document/d/1t-UVIO9Aq0TPUFHyo1nVscqaW1LoPuNhfLPitL8oeTs/edit?usp=sharing
> 
> `nap

To confirm what I see from looking at your data could you tell me if this
patch triggers a barf?

--- mem.c.orig  2019-05-18 20:04:26.707995000 +
+++ mem.c   2019-05-18 20:04:02.205119000 +
@@ -97,6 +97,7 @@
return (0);
}

+printf("mmio_rb_lookup: barf ENOENT");
return (ENOENT);
 }

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [vm-bhyve] adding existing vm instances to the vm-bhyve system

[Rodney W. Grimes]

> Hi,
> 
> How would one import a freebsd or linux vm currently running on a
> freebsd-12 server into vm-bhyve management? I can see lots of
> instructions for starting from fresh, but none for import of existing.
> Is this possible?
> 
> The guests are currently running as per
> https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/virtualization-host-bhyve.html
> in screen. Some guests are file-backed, others zvol-backed.
> 
> thanks,
> J.

This is possible, and not overly complex, but also non-trivial to do.
First some preliminary data collection and issue resolution needs to
be done.

a)  vm-bhyve assumes each vm lives in a directory by itself, and the
collection of vm's all share a common parent.  If your setup is
not like this you need to try and bring it to this form.

b)  vm-bhybe wants to, but does not have to, manage all your tap
devices, creating and destroying them as you go, sorting this
out between /etc/rc.conf methods and vm-bhyve methods can be
messy if you have done anything very fancy.


The mechanics once this much is done is a matter of writting a
fairly direct translation of your bhyve(8) command into a .conf
file that vm-bhyve uses.

Something like this for my
more fb-bld-13-amd64/fb-bld-13-amd64.conf
(Note:  The directory name and .conf file name must match for
vm-bhyve to work, which makes an easy way to make vms disappear
from its management by changing either name)

loader="bhyveload"
cpu=4
memory=2048M
network0_type="virtio-net"
network0_switch="trunked"
#disk0_name="/home/ISO/x/FreeBSD-13.0-CURRENT-amd64-20181213-r342020-disc1.iso"
#disk0_dev="custom"
#disk0_type="ahci-hd"
disk0_type="ahci-hd"
disk0_name="disk0.img"
wiredmem=1
network0_mac="58:9c:fc:00:ce:7f"

I picked this sample specifically as it shows how to refer to a "disk"
that is not in the home directory of the vm by using the _dev=custom
option, which you may very well need to do for some of your vms.

You do not need to put the *_mac stuff in there, vmbhyve well automagically
add those for you.

Personally I have migrated both my self managed bhyve stuff, and almost all of
my ESXi based FreeBSD vm's to vm-bhyve. 
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop

[Rodney W. Grimes]

> Victor Sudakov wrote:
> > 
> > I've noticed that it takes too long for Windows 2012 and 2016 servers
> > guests to shutdown when I issue "vm stopall," several minutes even.
> > 
> > Does vm-bhyve provide a way to power them off ungracefully if they would
> > not stop within a predefined time?
> > 
> > Or is there perhaps a way to tune the guest OSes to react more promptly
> > to an ACPI shutdown?
> > 
> > My primary concern is that my UPS and "apcupsd --kill-on-powerfail" give
> > me about 30 seconds of grace time before the power supply is cut out. It
> > is not sufficient for the bhyve box to shutdown shutdown properly
> > because it waits for the guests to shutdown.
> 
> For the present I think I've found a workaround not directly related to
> vm-bhyve.  I'm going to insert "service vm stop" into the doshutdown()
> procedure in /usr/local/etc/apcupsd/apccontrol before the actual
> ${SHUTDOWN}. It should give the VMs sufficient time to shutdown before
> the actual /sbin/shutdown is executed.

Make sure it does not do the waitall for pids or your going to
sit here waiting for that to complete for a long time, ie until
all vm's go to power off.

> > 
> > If you know of a better way to configure apcupsd so that it powers down
> > the UPS only after all the guest OSes and other daemons are safely down,
> > I'd appreciate that too.
> 
> Do you think the above hack would have any negative effects?

I think this is a reasonable approach given the current situation.

> -- 
> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve, sparse disk image and trim support?

[Rodney W. Grimes]

[ Charset UTF-8 unsupported, converting... ]
> Hi!
> 
> > Am 21.04.2019 um 19:01 schrieb Bjoern A. Zeeb 
> > :
> > Do we support this?  Has anyone worked on this?  How do people deal with 
> > these problems?
> 
> I?m actually surprised this is supported for some configurations - thanks
> for starting the thread. As for how others deal with this - e.g. VMware files
> never shrink, but there is a ?vacuuming? function that compresses the
> virtual disk while the VM is offline in the workstation products. For ESXi
> I think the only way is to actually copy the image file to a new one with
> vmkfstools.

The current state of affairs with respect to ESXi is expressed here:
https://www.codyhosterman.com/2016/11/whats-new-in-esxi-6-5-storage-part-i-unmap/

There was some experimental stuff back in 5.0 that was mostly broken.
 
And there is much more information out there to be read

> Used to use defrag and sdelete a lot in Windows VMs in the past, then
> ran the compaction ?

I am not sure, but at one time I thought ESXi would detect writes of
null blocks to thin provisioned disks and unmap those blocks on its
own, and that was basically how the guest tools cleanup disk space
thing worked, but I can not find that information on this today.

I do know that dd conv=sparse is one way to relaim such null
blocks from a FreeBSD host if you do
dd if=/dev/zero of=somejunkplace bs=1m
rm somejunkplace
inside your guest first, shut it down, then run a conv=sparse
you can get back host file system space.  Though the dd inside
the guest well usually fully inflate the disk image so be
careful!  You may need up to 2 times the volume size to do
these operations.

> Kind regards
> Patrick

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop

[Rodney W. Grimes]

-- Start of PGP signed section.
> Paul Vixie wrote:
> > 
> > Victor Sudakov wrote on 2019-04-22 19:43:
> > ...
> > >> And the implementation is pretty brutal:
> > >> # 'vm stopall'
> > >> # stop all bhyve instances
> > >> # note this will also stop instances not started by vm-bhyve
> > >> #
> > >> core::stopall(){
> > >>  local _pids=$(pgrep -f 'bhyve:')
> > >>
> > >>  echo "Shutting down all bhyve virtual machines"
> > >>  killall bhyve
> > >>  sleep 1
> > >>  killall bhyve
> > >>  wait_for_pids ${_pids}
> > >> }
> > 
> > yow.
> 
> To be sure, I was unable to find the above code (as is) in
> /usr/local/lib/vm-bhyve/vm-* (the vm-bhyve port 1.3.0). It may be that
> something more intelligent is happening in a more recent version, like a
> sequential shutdown. However, "kill $pid; sleep 1; kill $pid" seems to
> be still present.

I probably pulled that from old code, pulled from:
vm-bhyve-1.2.3 Management system for bhyve virtual machines

> 
> > 
> > >>
> > >> I wonder what the effect of the second kill is,
> > >> that seems odd.
> > > 
> > > Indeed.
> > 
> > the first killall will cause each client OS to see a soft shutdown 
> > signal. the sleep 1 gives them some time to flush their buffers. the 
> > second killall says, time's up, just stop.
> > 
> > i think this is worse than brutal, it's wrong. consider freebsd's own 
> > work flow when trying to comply with the first soft shutdown it got:
> > 
> > https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220
> > 
> > this has bitten me more than once, because using "pageins" as a proxy 
> > for "my server processes are busy trying to synchronize their user mode 
> > state" is inaccurate. i think _any_ continuing I/O should be reason to 
> > wait the full 60 seconds.
> 
> Would it be beneficial to just hack /usr/local/lib/vm-bhyve/vm-* ?

One can always hack experiments, my vm-bhyve is fairly diverged
from the release bits as I have done stuff to it so that it
better meets my needs.  Most of that is not submittable as upstream
changes, some I should really sort out and try to push up.

Some is due to local changes to bhyve that are not mainstream
and thus are not (yet) applicable.

> > and so i think the "sleep 1" above should be a "sleep 65".
> > 
> > > What is needed in vm-bhyve is the feature that if ACPI does not stop the
> > > guest for a predefined period of time, the guest is powered off.
> > 
> > i agree with this.
> 
> Will you please support the bug report: 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237479

More powerfull would be if we could come up with some
patches against 1.3.0 that effected some of the changes
we desire.  And some more investigation as to just how
the guests are handling this ACPI shutdown event.  What
might be wrong for FreeBSD might be right for windows?
 
Does the ACPI spec saying anything about hitting the
power down button twice within 1 second for example?

> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop

[Rodney W. Grimes]

> Victor Sudakov wrote on 2019-04-22 19:43:
> ...
> >> And the implementation is pretty brutal:
> >> # 'vm stopall'
> >> # stop all bhyve instances
> >> # note this will also stop instances not started by vm-bhyve
> >> #
> >> core::stopall(){
> >>  local _pids=$(pgrep -f 'bhyve:')
> >>
> >>  echo "Shutting down all bhyve virtual machines"
> >>  killall bhyve
> >>  sleep 1
> >>  killall bhyve
> >>  wait_for_pids ${_pids}
> >> }
> 
> yow.
> 
> >>
> >> I wonder what the effect of the second kill is,
> >> that seems odd.
> > 
> > Indeed.
> 
> the first killall will cause each client OS to see a soft shutdown 
> signal. the sleep 1 gives them some time to flush their buffers. the 
> second killall says, time's up, just stop.
> 
> i think this is worse than brutal, it's wrong. consider freebsd's own 
> work flow when trying to comply with the first soft shutdown it got:
> 
> https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220

Interesting, more interesting would be to dig out the
path that the system takes when it gets teh ACPI shutdown
event.  That is not going to end up in sbin/reboot is it?

Doesnt that run from within init itself?

I find in the init man page this:
 The init utility will terminate all possible processes (again, it will
 not wait for deadlocked processes) and reboot the machine if sent the
 interrupt (INT) signal, i.e. ``kill -INT 1''.  This is useful for
 shutting the machine down cleanly from inside the kernel or from X when
 the machine appears to be hung.

So my guess is that sending an ACPI shutdown event to the VM ends up
sending a -INT to init.  Looking inside init, it seems to end up
execing a shell runing /etc/rc.shutdown.

I do not know if the ACPI event is then blocked in the kernel so
the second one is pointless, but I believe once we enter into
-INT processing that signal is ignored, so infact this second
signal is actaully OK.  But I am not 100% sure on this, yet.


> this has bitten me more than once, because using "pageins" as a proxy 
> for "my server processes are busy trying to synchronize their user mode 
> state" is inaccurate. i think _any_ continuing I/O should be reason to 
> wait the full 60 seconds.
> 
> and so i think the "sleep 1" above should be a "sleep 65".

I think the sleep and the second signal are near pointless.  If
we have a race in ACPI processing we need to fix that.

> > What is needed in vm-bhyve is the feature that if ACPI does not stop the
> > guest for a predefined period of time, the guest is powered off.
> 
> i agree with this.

There is a PR up,
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237479
please add comments there if you feel so inclined.

> -- 
> P Vixie

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop

[Rodney W. Grimes]

-- Start of PGP signed section.
> Rodney W. Grimes wrote:
> > -- Start of PGP signed section.
> > > Jason Tubnor wrote:
> > > > On Sun., 21 Apr. 2019, 2:51 pm Victor Sudakov,  
> > > > wrote:
> > > > 
> > > > >
> > > > >
> > > > > Does vm-bhyve provide a way to power them off ungracefully if they 
> > > > > would
> > > > > not stop within a predefined time?
> > > > >
> > > > 
> > > > You'd have to do your own checking but to force an instant power off of 
> > > > a
> > > > guest simply execute:
> > > > 
> > > > vm poweroff {guest}
> > > 
> > > I don't need an *instant* poweroff, I need a poweroff only if a VM would
> > > not stop gracefully within a predefined time.
> > > 
> > > Of course this should not be done manually but from the rc.d/vm script
> > > on shutdown.
> > 
> > vm shutdown {guest} &
> > (sleep N && vm poweroff {guest}) &
> > 
> > A bit hackish, but that should work, if the guest has
> > shutdown by the time you get to the poweroff an error
> > might occur.  This spawns a few processes, but is
> > non blocking and highly parallel.
> 
> Thank you, I might give it a try though it would involve hacking the
> rc.d/vm script.
> 
> >  
> > One thing I noted, you said rc.d/vm, so I am assuming your actually
> > using the vm-bhyve port, 
> 
> Yes, I thought that was pretty obvious from the Subject line.

Subject line disappears off the screen once a screen full of
email gets displayed :-)

> > in which case it has a sequential shutdown
> > with time delays between each guest.  vm_delay specifically, and it
> > shuts vm's down in reverse order of the startup.
> > 
> > That is more likely your problem in that your sending these acpi
> > shutdown requests one at a time, and they should be broadcast in
> > the "power going out" case.
> 
> Whence is the idea that "vm stopall" does a sequential shutdown? What sense
> would that make? 

Well I am not sure that vm-bhyve does this, but esxi certainly does,
and it is a royal PITA to deal with sometimes.  It does make sense
in some aspects, do not want the database server going offline before
all the clients go down do we?  Kinda makes for a bunch of nonsense
errors logged due to missing server.  I kinda like my virtual routers
and firewalls to keep going tell the end too.

This is all YMMV situations though.

> A sequential startup does make sense but a sequential shutdown? Useless
> I think. The man page says that 

For you perhaps useless, but that rarely rules out that there may be
a totally valid and usefull case.

> stopall
>  Stop all running virtual machines.  This sends a stop command to
>  all bhyve(8) instances, regardless of whether they were starting
>  using vm or not.

And the implementation is pretty brutal:
# 'vm stopall'
# stop all bhyve instances
# note this will also stop instances not started by vm-bhyve
# 
core::stopall(){
local _pids=$(pgrep -f 'bhyve:')

echo "Shutting down all bhyve virtual machines"
killall bhyve
sleep 1
killall bhyve
wait_for_pids ${_pids}
}

I wonder what the effect of the second kill is,
that seems odd.  Almost like you might cause more
issues than you solve as now you already have a
vm in what should be ACPI shutdown process.

Also this killall operation probably puts a high stress
on disk i/o as you just woke up and made all the vm's
get busy all at once and your going to basically thrash
on every single resource they are using (yet another reason
you may actually want to serialize these shutdown operations.)

IIRC windows, especially newer ones, do a boat load of work
on a shutdown unless they are told to shutdown quickly.  Ie
they even try to apply pending updates and such, that could
be part of why you see windows vm's lagging around.

You may also want to: Disable Clear page file on shutdown
that is a windows thing, if you have a huge page file that
can do a LOT of io, if you have a few windows vm's on the
same spindle and try to stop them all at once your going to
trash that disk for much longer than you need.

> > 
> > It may be possile to adjust vm_delay to 0 and have that be better,
> > though I have not locked at the code.  You may also wish to discuss
> > the issue with the vm-bhyve maintainer and maybe a "lights out"
> > procedure needs to be added.
> 
> Even if there is a single Windows 2012 or 2016 guest running, it takes
> the vm several minutes to transition to the "stopped" state.

You might nuke the wait_for_pids?

> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop

[Rodney W. Grimes]

-- Start of PGP signed section.
> Jason Tubnor wrote:
> > On Sun., 21 Apr. 2019, 2:51 pm Victor Sudakov,  wrote:
> > 
> > >
> > >
> > > Does vm-bhyve provide a way to power them off ungracefully if they would
> > > not stop within a predefined time?
> > >
> > 
> > You'd have to do your own checking but to force an instant power off of a
> > guest simply execute:
> > 
> > vm poweroff {guest}
> 
> I don't need an *instant* poweroff, I need a poweroff only if a VM would
> not stop gracefully within a predefined time.
> 
> Of course this should not be done manually but from the rc.d/vm script
> on shutdown.

vm shutdown {guest} &
(sleep N && vm poweroff {guest}) &

A bit hackish, but that should work, if the guest has
shutdown by the time you get to the poweroff an error
might occur.  This spawns a few processes, but is
non blocking and highly parallel.
 
One thing I noted, you said rc.d/vm, so I am assuming your actually
using the vm-bhyve port, in which case it has a sequential shutdown
with time delays between each guest.  vm_delay specifically, and it
shuts vm's down in reverse order of the startup.

That is more likely your problem in that your sending these acpi
shutdown requests one at a time, and they should be broadcast in
the "power going out" case.

It may be possile to adjust vm_delay to 0 and have that be better,
though I have not locked at the code.  You may also wish to discuss
the issue with the vm-bhyve maintainer and maybe a "lights out"
procedure needs to be added.

> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
Regards,
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Windows 2019 server

[Rodney W. Grimes]

> On Thu, Apr 11, 2019 at 10:10:03PM -0500, Dustin Marquess wrote:
> > It's worth a shot at least to see if it works!
> > 
> > But ya, basically the same as mine:
> > 
> > bhyve -A -H -P -w -S -u -c sockets=1,cores=4,threads=2 -m ${RAM} \
> > -s 0,hostbridge \
> > -s 3,nvme,/dev/zvol/tank/vm/win2019 \
> > -s 4,ahci-cd,${VMROOT}/win2019/empty.iso,nocache,ro \
> > -s 5,passthru,130/0/0 \
> > -s 10,virtio-net,tap2 \
> > -s 20,virtio-rnd \
> > -s 31,lpc \
> > -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd ${VM}
> > 
> > -Dustin
> 
> I will try a bit of a mix.  Also  Do the same for the Linux emulations 
> such as Scientific, Fedora, Centos, Debian and Ubuntu and ORacle?
> 
> BTW  rational for the -w , your -c content your passthru and the virtio-rnd?

The -c context is done most likely for license reasons,
without specifying topology the default for -c 8 would
be 8 sockets of 1 core of 1 thread, that would require
a high end server win2019 license.  By building this
as a 8 threads in a single socket you get to use a
single socket windows license.  This is one of the major
reasons that I added the complex topology specification
to the -c option.

> > On Thu, Apr 11, 2019 at 5:54 PM The Doctor  wrote:
> > >
> > > On Thu, Apr 11, 2019 at 05:18:10PM -0500, Dustin Marquess wrote:
> > > > I'm not sure if 12.0 has bhyve nmve support, but 10 & 2019 both seem
> > > > to run MUCH faster when using nvme compared to ahci-hd.
> > > >
> > > > -Dustin
> > > >
> > >
> > > Here is the script I use to start the Windows server
> > >
> > >
> > > #!/bin/sh
> > > bhyve -c 4 -s 0,hostbridge -s 3,ahci-hd,/usr/vm/images/windows2019st  -s 
> > > 10,virtio-net,tap3 -s 31,lpc -l com1,/dev/nmdm9A -l com2,/dev/nmdm10A -s 
> > > 29,fbuf,tcp=0.0.0.0:5958,w=1024,h=768,wait -l 
> > > bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -m 4G -H -w 
> > > windows2019st &
> > >
> > > So I should change ahci-cd to nvme?
> > >
> > >
> > > > On Thu, Apr 11, 2019 at 4:40 PM The Doctor via freebsd-virtualization
> > > >  wrote:
> > > > >
> > > > > Seems to be running slow on FreeBSD 12.0 p3.
> > > > >
> > > > > Just wondering if there are known issues.
> > > > > --
> > > > > Member - Liberal International This is doctor@@nl2k.ab.ca Ici 
> > > > > doctor@@nl2k.ab.ca
> > > > > Yahweh, Queen & country!Never Satan President Republic!Beware 
> > > > > AntiChrist rising!
> > > > > https://www.empire.kred/ROOTNK?t=94a1f39b  Look at Psalms 14 and 53 
> > > > > on Atheism
> > > > > Alberta on 16 April 2019, do not vote UCP, FCP nor NDP!
> > > > > ___
> > > > > freebsd-virtualization@freebsd.org mailing list
> > > > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > > > > To unsubscribe, send any mail to 
> > > > > "freebsd-virtualization-unsubscr...@freebsd.org"
> > >
> > > --
> > > Member - Liberal International This is doctor@@nl2k.ab.ca Ici 
> > > doctor@@nl2k.ab.ca
> > > Yahweh, Queen & country!Never Satan President Republic!Beware AntiChrist 
> > > rising!
> > > https://www.empire.kred/ROOTNK?t=94a1f39b  Look at Psalms 14 and 53 on 
> > > Atheism
> > > Alberta on 16 April 2019, do not vote UCP, FCP nor NDP!
> 
> -- 
> Member - Liberal International This is doctor@@nl2k.ab.ca Ici 
> doctor@@nl2k.ab.ca
> Yahweh, Queen & country!Never Satan President Republic!Beware AntiChrist 
> rising!
> https://www.empire.kred/ROOTNK?t=94a1f39b  Look at Psalms 14 and 53 on Atheism
> Alberta on 16 April 2019, do not vote UCP, FCP nor NDP!
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: running FreePBX SNG7 Official Distro

[Rodney W. Grimes]

> Victor Sudakov wrote:
> > > 
> > > As I stated earlier bhyve is missing percistant efi variables,
> > > and that is most likely the reason that VirtualBox just works
> > > and bhyve does not.
> > > 
> > > Probably you well find in your VirtualBox directory a
> > > file that is used to store efivars, that is where the
> > 
> > I'll look into the VirtualBox directory tomorrow and report here. 
> 
> I searched through my disk and was unable to find a persistant efivars
> storage in my VirtualBox 6.0 installation. 
> 
> A Google search reveals some articles (rather dated I must admit)
> stating that VirtualBox does not support NVRAM emulation for storing efi
> variables: 
> 
> https://www.virtualbox.org/ticket/14279
> https://forums.virtualbox.org/viewtopic.php?t=61970 

My quick search turns up:
https://www.virtualbox.org/svn/vbox/trunk/src/VBox/Devices/EFI/DevEFI.cpp

aka source code thet implements efivars stored in nvram.

> they recommend using startup.nsh instead.  I wonder if bhyve's efi
> implementation supports startup.nsh.

I believe that is how the alternate boot selector someone
pointed at gets hooked in.

> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
> 2:5005/49@fidonet http://vas.tomsk.ru/

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: running FreePBX SNG7 Official Distro

[Rodney W. Grimes]

> On Sun, Apr 7, 2019 at 11:15 AM Victor Sudakov  wrote:
> 
> > I'll look into the VirtualBox directory tomorrow and report here. I was
> > under the impression that efivars are stored in a configuration file in
> > the EFI partition but I was probably wrong, they are kept in NVRAM
> > somewhere, like BIOS settings, and not on a disk.
> >
> > --
> > Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
> > 2:5005/49@fidonet http://vas.tomsk.ru/
> 
> I just wanted to confirm that this is indeed an EVIVARS problem,
> because through CBSD (which uses Refind [1]) FreePBX 1805-2
> boots without any problems. Therefore, as Rodney said, you can try
> using a third-party EFI boot manager.
> And just wait until someone finds the time to add support for UEFI
> VARS in bhyve.

Thank you for the analysis and confirmation!

> [1] - http://www.rodsbooks.com/refind/
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: running FreePBX SNG7 Official Distro

[Rodney W. Grimes]

> Victor Sudakov wrote:
> > > > > > I can guess that it looks for a FAT16 partition in the GPT with the 
> > > > > > type
> > > > > > "efi" but the rest is a mystery for me. Why is it trying to find
> > > > > > "grubx64.efi" and not the default "boot64.efi" (which is present), 
> > > > > > for
> > > > > > example?
> > > > > 
> > > > > I suspect that what ever guest you installed installed something
> > > > > else someplace, either within the eft partition, or possibly in
> > > > > the MBR?
> > > > 
> > > > Do you mean to say, the guest installing something else someplace can
> > > > influence the boot sequence of bhyve efi?
> > > 
> > > The guest created all of the bits on that zvol,
> > > it can influence many things.  There is probably a tiny initial
> > > stub that efi loads that has this bath to grubx64.efi codded in
> > > it and that is what is causing this issue.
> > 
> > It is very important to find and debug it because Oracle VirtualBox in
> > UEFI mode installs and runs this guest just fine. So it must be some
> > issue in bhyve itself.
> > 
> > Here is the complete archive of everything the guest created in the EFI
> > partition: http://admin.sibptus.ru/~vas/freepbx.tar.gz
> > can you find those confusing bits?
> 
> I got it! bhyve does the right thing: it tries to boot BOOTX64.EFI, but
> BOOTX64.EFI  makes it look for grubx64.efi. So BOOTX64.EFI must be some
> kind of chain loader.

And it brobably tries to read a efivariable, and if that variable
is not set it defaults to grubx64.efi.  This bootx64.efi is something
the guest installed into the EFI partition, hence my assertion that
the issue is with something the guest installed is some what valid.

There are some 3rd party EFI boot managers that might help
resolve this problem, or simply use the work around that
I provided earlier until we can get efivars working in
bhyve.


> Watch the interactive session below.  It does not however mean that there is
> nothing to fix. As I said Oracle VirtualBox in UEFI mode installs and runs 
> this
> guest just fine.
> 
> FS0:\> cd EFI
> FS0:\EFI\> ls
> Directory of: FS0:\EFI\
> 04/04/2019  15:53  2,048  .
> 04/04/2019  15:53  0  ..
> 04/04/2019  16:26  2,048  centos
> 04/06/2019  04:19  2,048  BOOT
>   0 File(s)   0 bytes
>   4 Dir(s)
> FS0:\EFI\> cd BOOT
> FS0:\EFI\BOOT\> ls
> Directory of: FS0:\EFI\BOOT\
> 04/04/2019  16:18  2,048  .
> 04/04/2019  16:18  2,048  ..
> 08/31/2017  21:30   1,296,176  BOOTX64.EFI
> 08/31/2017  21:30  79,048  fbx64.efi
>   2 File(s)   1,375,224 bytes
>   2 Dir(s)
> FS0:\EFI\BOOT\> BOOTX64.EFI
> Failed to set MokListRT: Invalid Parameter
> Failed to open \EFI\BOOT\grubx64.efi - Not Found
> Failed to load image \EFI\BOOT\grubx64.efi: Not Found
> start_image() returned Not Found
> FS0:\EFI\BOOT\> 
> 
> -- 
> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
> 2:5005/49@fidonet http://vas.tomsk.ru/

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: running FreePBX SNG7 Official Distro

[Rodney W. Grimes]

> Rodney W. Grimes wrote:
> > > > 
> > > > You can usually use the host by doing mdconfig -f 
> > > 
> > > Unfortunately mdconfig does not work with zvols:
> > > 
> > > root@vas:~ # mdconfig -a -f /dev/zvol/d02/vm/freepbx/disk0 
> > > mdconfig: /dev/zvol/d02/vm/freepbx/disk0 is not a regular file
> > 
> > If its a zvol cant you just do
> > gpart show /dev/zvol/d02/vm/freepbx/disk0
> > 
> > and 
> > mount -t msdosfs /dev/zvol/d02/vm/freepbx/disk0p2
> 
> No I can't if the zvol is in the "volmode=dev" mode which is the default. 
> 
> This is the default for a reason: it's no good exposing scores of always
> coming and going guest geoms to the host system. I think you can even
> get a conflict of labels or something like that one day.

So it may take a few more commands but it should be
possible to do this from the host side using host
side tools without having to boot a guest to make
these corrections.

> > > > > Moreover, I waited (for a long time!) for the EFI interactive shell
> > > > > prompt and with a few commands:
> > > > 
> > > > Yes, the timeout is very long, and I do not know that we
> > > > document anyplace that if you wait long enough at a failed
> > > > boot you do get a EFI shell prompt eventually.
> > > 
> > > Can I press some key to escape to the EFI shell?
> > Not that I am aware of.
> 
> It's a major problem! There must be a well-known way to break the boot
> sequence any time and enter the EFI shell.

Agreed, hopefully those working on edk2 take note and either
chime in with what that way is, or create a bug and track
so that someone may fix this issue.

> > > > > I can guess that it looks for a FAT16 partition in the GPT with the 
> > > > > type
> > > > > "efi" but the rest is a mystery for me. Why is it trying to find
> > > > > "grubx64.efi" and not the default "boot64.efi" (which is present), for
> > > > > example?
> > > > 
> > > > I suspect that what ever guest you installed installed something
> > > > else someplace, either within the eft partition, or possibly in
> > > > the MBR?
> > > 
> > > Do you mean to say, the guest installing something else someplace can
> > > influence the boot sequence of bhyve efi?
> > 
> > The guest created all of the bits on that zvol,
> > it can influence many things.  There is probably a tiny initial
> > stub that efi loads that has this bath to grubx64.efi codded in
> > it and that is what is causing this issue.
> 
> It is very important to find and debug it because Oracle VirtualBox in
> UEFI mode installs and runs this guest just fine. So it must be some
> issue in bhyve itself.

As I stated earlier bhyve is missing percistant efi variables,
and that is most likely the reason that VirtualBox just works
and bhyve does not.

> Here is the complete archive of everything the guest created in the EFI
> partition: http://admin.sibptus.ru/~vas/freepbx.tar.gz
> can you find those confusing bits?

Probably you well find in your VirtualBox directory a
file that is used to store efivars, that is where the
difference occurs.

> The standard procedure should be as follows:
> 
> Automated detection relies on standardized file paths to the OS
> loader, with the path varying depending on the computer architecture.
> The format of the file path is defined as
> /EFI/BOOT/BOOT.EFI; for
> example, the file path to the OS loader on an x86-64 system is
> /efi/BOOT/BOOTX64.EFI and efi\boot\bootaa64.efi on ARM64 architecture. 
> 
> Nothing about grub*.efi. But only bhyve is confused, VirtualBox is not.

bhyve is not really confused, it is simply missing a feature
that this guest depends on for its boot procedures.  This is
a well known miss-feature, but I do not know of anyone actively
working on fixing it.

> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
> 2:5005/49@fidonet http://vas.tomsk.ru/
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: running FreePBX SNG7 Official Distro

[Rodney W. Grimes]

> Rodney W. Grimes wrote:
> > > 
> > > [dd]
> > > 
> > > > > 
> > > > > root@mfsbsd:~ # find /mnt/ -name grubx64.efi
> > > > > /mnt/EFI/centos/grubx64.efi
> > > > > 
> > > > > Who is to blame, bhyve or FreePBX's installer?
> > > > > 
> > > > > How can I tell bhyve's UEFI loader to look for grubx64.efi in a
> > > > > different place? Or look for a different loader?
> > > > > 
> > > > > Who says that the image to load should be "\EFI\BOOT\grubx64.efi" and
> > > > > not "\EFI\BOOT\BOOTX64.EFI" for example?
> > > > 
> > > > I can not quickly answer that, but lets try the short quick fix
> > > > and simply copy this file to the right place and see if that
> > > > gets you up and running. 
> > > 
> > > Yes, copying grubx64.efi to "\EFI\BOOT\" does get the guest up and
> > > running (I used mfsbsd from a different VM to manipulate the EFI
> > > partition).
> > 
> > You can usually use the host by doing mdconfig -f 
> 
> Unfortunately mdconfig does not work with zvols:
> 
> root@vas:~ # mdconfig -a -f /dev/zvol/d02/vm/freepbx/disk0 
> mdconfig: /dev/zvol/d02/vm/freepbx/disk0 is not a regular file

If its a zvol cant you just do
gpart show /dev/zvol/d02/vm/freepbx/disk0

and 
mount -t msdosfs /dev/zvol/d02/vm/freepbx/disk0p2

> 
> > > Moreover, I waited (for a long time!) for the EFI interactive shell
> > > prompt and with a few commands:
> > 
> > Yes, the timeout is very long, and I do not know that we
> > document anyplace that if you wait long enough at a failed
> > boot you do get a EFI shell prompt eventually.
> 
> Can I press some key to escape to the EFI shell?
Not that I am aware of.

> > >  Shell> fs0
> > >  FS0:\> cd \EFI\centos
> > >  FS0:\EFI\centos\> grubx64.efi   
> > > 
> > > I also managed to boot the guest OS all right.
> > > 
> > > But naturally, the latter fix worked till next reboot only, I don't know
> > > how to save the new EFI setup in the guest's configuration.
> > 
> > My recommedation at this time would be to simply copy grubx64.efi
> > to the right place and leave it there so that it just boots without
> > any other change.
> 
> That's what I have done for now.
> 
> > > 
> > > The hardware UFI BIOSes I've seen so far (not many, I must admit)
> > > permitted me to save which efi binary I would prefer to boot next time.
> > 
> > That is done with an efivar, as it stands right now bhyve efi has
> > no persistant variable storage, a feature that needs to be implemented.
> 
> I see.
> 
> [dd]
> 
> > 
> > > I can guess that it looks for a FAT16 partition in the GPT with the type
> > > "efi" but the rest is a mystery for me. Why is it trying to find
> > > "grubx64.efi" and not the default "boot64.efi" (which is present), for
> > > example?
> > 
> > I suspect that what ever guest you installed installed something
> > else someplace, either within the eft partition, or possibly in
> > the MBR?
> 
> Do you mean to say, the guest installing something else someplace can
> influence the boot sequence of bhyve efi?

The guest created all of the bits on that zvol,
it can influence many things.  There is probably a tiny initial
stub that efi loads that has this bath to grubx64.efi codded in
it and that is what is causing this issue.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: running FreePBX SNG7 Official Distro

[Rodney W. Grimes]

-- Start of PGP signed section.
> Rodney W. Grimes wrote:
> 
> [dd]
> 
> > > 
> > > root@mfsbsd:~ # find /mnt/ -name grubx64.efi
> > > /mnt/EFI/centos/grubx64.efi
> > > 
> > > Who is to blame, bhyve or FreePBX's installer?
> > > 
> > > How can I tell bhyve's UEFI loader to look for grubx64.efi in a
> > > different place? Or look for a different loader?
> > > 
> > > Who says that the image to load should be "\EFI\BOOT\grubx64.efi" and
> > > not "\EFI\BOOT\BOOTX64.EFI" for example?
> > 
> > I can not quickly answer that, but lets try the short quick fix
> > and simply copy this file to the right place and see if that
> > gets you up and running. 
> 
> Yes, copying grubx64.efi to "\EFI\BOOT\" does get the guest up and
> running (I used mfsbsd from a different VM to manipulate the EFI
> partition).

You can usually use the host by doing
mdconfig -f 
gpart show md0  # Assuming mdconfig was unit 0
mount -t msdosfs /dev/md0p /mnt  #The n comes from gpart, you want the efi 
partition

Now you can manipulate /mnt/ all you want
umount /mnt
mdconfig -d -u 0# Assuming mdconfig was unit 0

> Moreover, I waited (for a long time!) for the EFI interactive shell
> prompt and with a few commands:

Yes, the timeout is very long, and I do not know that we
document anyplace that if you wait long enough at a failed
boot you do get a EFI shell prompt eventually.

> 
>  Shell> fs0
>  FS0:\> cd \EFI\centos
>  FS0:\EFI\centos\> grubx64.efi   
> 
> I also managed to boot the guest OS all right.
> 
> But naturally, the latter fix worked till next reboot only, I don't know
> how to save the new EFI setup in the guest's configuration.

My recommedation at this time would be to simply copy grubx64.efi
to the right place and leave it there so that it just boots without
any other change.

> 
> The hardware UFI BIOSes I've seen so far (not many, I must admit)
> permitted me to save which efi binary I would prefer to boot next time.

That is done with an efivar, as it stands right now bhyve efi has
no persistant variable storage, a feature that needs to be implemented.

> > That would also tell us that we have
> > what is actually a common efi system failure problem in that
> > stuff looks in the wrong place.
> 
> It seems so.
> 
> >  I have read many an install
> > instruction that just says copy this file to these too places
> > as some bioses look for it in one place and others look for it
> > someplace else.
> 
> I would very much appreciate a link to some such instruction about
> uefi-edk2-bhyve: namely how and where it looks for what on boot, and if I
> can create a menu for example, or change its startup procedure.

I do not know where that information is, others may be able to
fill in more details.

> I can guess that it looks for a FAT16 partition in the GPT with the type
> "efi" but the rest is a mystery for me. Why is it trying to find
> "grubx64.efi" and not the default "boot64.efi" (which is present), for
> example?

I suspect that what ever guest you installed installed something
else someplace, either within the eft partition, or possibly in
the MBR?

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: emulated PCI device BAR size

[Rodney W. Grimes]

> On Sat, Mar 23, 2019 at 6:44 PM Jason Tubnor  wrote:
> >
> >
> >
> > On Sun., 24 Mar. 2019, 6:35 am Chuck Tuffli,  wrote:
> >>
> >> On Mon, Mar 18, 2019 at 10:15 AM Chuck Tuffli  wrote:
> >>
> >> >
> >> > So my question is how to best fix this. The easiest would be to add a
> >> > check to the BAR size calculation in pci_nvme.c along the lines of:
> >> > #define NVME_MMIO_SPACE_MIN(1 << 14)
> >> > ...
> >> > pci_membar_sz = MAX( ,
> >> > NVME_MMIO_SPACE_MIN);
> >>
> >> I went down this route and have a patch up for review
> >> (https://reviews.freebsd.org/D19676) if anyone is interested.
> >>
> >> --chuck
> >
> >
> > I've built and installed this patch on FreeBSD 
> > 12.0-STABLE-amd64-20190314-r345117.  Testing occurred against Windows 10 
> > Pro guest with a presented NVMe bhyve device and all IO transactions worked 
> > without issue.
> >
> > Can this be committed to head and MFD to 12-STABLE.  Thanks!
> 
> Committed with r345956. MFC in 2 weeks if nothing pops up. Thanks
> again for all your testing!

A small prod Chuck you should of mentioned Jason in the
commit message for his testing work.  I know it is easy
to forget this, but it is the right thing to do.

Jason,
  I'll thank you here, in a more personal maner, for working with
Chuck on getting this fixed.

Thanks,
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: running FreePBX SNG7 Official Distro

[Rodney W. Grimes]

-- Start of PGP signed section.
> Victor Sudakov wrote:
> > 
> > Has anyone tried to run FreePBX under bhyve? That's what I get trying to
> > start the vm after a successful automatic install from the ISO image:
> > 
> > Boot Failed. EFI DVD/CDROM
> > Failed to set MokListRT: Invalid Parameter
> > Failed to open \EFI\BOOT\grubx64.efi - Not Found
> > Failed to load image \EFI\BOOT\grubx64.efi: Not Found
> > start_image() returned Not Found
> > Boot Failed. EFI Misc Device
> 
> Below are the partitions the automatic installer has created (looking at
> them from another vm):
> 
> root@mfsbsd:~ # gpart show vtbd1
> =>  34  41942973  vtbd1  GPT  (20G)
> 34  2014 - free -  (1.0M)
>   2048186368  1  efi  (91M)
> 188416   4096000  2  ms-basic-data  (2.0G)
>4284416  37654528  3  linux-lvm  (18G)
>   41938944  4063 - free -  (2.0M)
> 
> If I "mount_msdosfs /dev/vtbd1p1 /mnt/" I see that grubx64.efi is not
> where bhyve expects to find it:
> 
> root@mfsbsd:~ # find /mnt/ -name grubx64.efi
> /mnt/EFI/centos/grubx64.efi
> 
> Who is to blame, bhyve or FreePBX's installer?
> 
> How can I tell bhyve's UEFI loader to look for grubx64.efi in a
> different place? Or look for a different loader?
> 
> Who says that the image to load should be "\EFI\BOOT\grubx64.efi" and
> not "\EFI\BOOT\BOOTX64.EFI" for example?

I can not quickly answer that, but lets try the short quick fix
and simply copy this file to the right place and see if that
gets you up and running.  That would also tell us that we have
what is actually a common efi system failure problem in that
stuff looks in the wrong place.  I have read many an install
instruction that just says copy this file to these too places
as some bioses look for it in one place and others look for it
someplace else.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve with UEFI edk2-stable201903 and CSM support - debugging the crash

[Rodney W. Grimes]

> I've spent some time trying to figure out where and why the UEFI 2.70 
> firmware is crashing when built with CSM support under Bhyve.
> 
> The first thing I noticed is that the code that builds the E820 table 
> doesn't appear to be finding the location of the EBDA at 0x40E: the 
> first entry should be something like 0x0 to 0x4cf00, but is 0-0 instead.

Since your building a bios there should be a table some
place in some header or c file that builds the EBDA and
puts a pointer to it at 0x40E.  This should be someplace
in the CSM legacy support stuff since the EBSA is a legacy
bios thingy.

You may want to look at the prior port of UEFI/CSM and see
how this was done in that code, it may be a hard coded hack.

> E820[ 0]: 0x?? 0  0x?? 0, Type = 0x1
> E820[ 1]: 0x?? 0  0x?? A, Type = 0x2
> E820[ 2]: 0x?? E  0x? 10, Type = 0x2
> E820[ 3]: 0x? 10  0x? 406000, Type = 0x1
> 
> ...
> 
> 
> As an aside, I'm working on a patch to submit upstream to fix the output 
> so it pads with zeros instead of spaces.
> 
> 
> I've also found that as was probably pretty obvious the system is 
> crashing when trying to call into the CSM to initialize it. It gets as 
> far as calling AsmThunk16 (AsmThunk.nasm), but never appears to reach 
> CsmMain in BhyveCsm16.c. I'll see if I can work on it some more to 
> narrow down the error.

Well I would think if the legacy memory region table is bogus
that would be a good start point to fix.

> 
> Another thing: the GNUmakefile in OvmfPkg/Csm/BhyveCsm16 has "CC ?= 
> gcc", but it appears the build system already sets CC to "cc", which 
> uses the system's clang compiler, which can't compile the assembly code.
> 
> I don't know if we might want to either hard-code CC to "gcc" for now, 
> or have users create a 'cc' symlinks in BaseTools/Bin/FreeBSD-amd64 ?

I would firmcode it, if CC=cc then CC=gcc, this would allow a person
to override CC with something else and still have all the other stuff
work out, except for this specific point that does infact need gcc.

> Rebecca Cran
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Updating uefi-edk2-bhyve

[Rodney W. Grimes]

> 
> On 3/30/19 2:37 PM, Rodney W. Grimes wrote:
> >> On 3/30/19 1:08 AM, Rodney W. Grimes wrote:
> >>>
> >>> So it is a edk2 implementation, and can you point me at
> >>> the bhyve version of it?  The Qemu/KVM version is rather
> >>> not relavent to this discussion as that would not work due
> >>> to core vmm implementation differences.
> 
> 
> Yes, OVMF is part of the TianoCore EDK2. OVMF/KVM is relevant in my 
> opinion because the BHYVE UEFI firmware is a port of OVMF. In fact, with 
> Scott's changes there's no longer a BhyvePkg, but our copy of OvmfPkg 
> works with Bhyve. The code can be found at 
> https://gitlab.com/scott-ph/edk2/tree/wip/2019-03/v2-bhyve-rebase-edk2-stable201903
>  
> .
> 
> 
> > ^^^ This is the critical version, this is the
> > version of the SMBIOS/dmitables implemented by the bios.
> 
> 
> I think you're confused: I wasn't asking about updating the version of 
> _SMBIOS_ that Bhyve uses, just the BIOS version and release date that 
> are shown in the BIOS section of the SMBIOS data.

Ack, I get your queston now, and my answer is:

Yes if Scott updated the edk2 port the bios version in that 
should defanitly be bumped.

What SMBIOS version is it reporting, out of curosity as I need
to do some work on that part of it with respect to my VM_MAXCPU
changes.   (Three reviews process, expect 3 to 5 more before
work is completed.)

> Rebecca Cran
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Updating uefi-edk2-bhyve

[Rodney W. Grimes]

> On 3/30/19 1:08 AM, Rodney W. Grimes wrote:
> >
> > I do not know what the above is, could you elaborate for me?
> > What is OVMF and what is its relation to bhyve?
> 
> 
> OVMF is the Open Virtual Machine Firmware: it's open source (BSD 
> licensed) firmware for virtual machines. Currently, that means Qemu/KVM. 
> It has been ported to run under Bhyve too.

So it is a edk2 implementation, and can you point me at
the bhyve version of it?  The Qemu/KVM version is rather
not relavent to this discussion as that would not work due
to core vmm implementation differences.

> > I have to assume this is with ed2k loaded, but I do not
> > know you are showing me the SMBIOS string value or some
> > other bios version value.  THere are compliance levels
> > associated with SMBIOS.
> 
> 
> I guess you mean edk2 (efi development kit 2), not ed2k?
Yes, I have been in Prague for 2 weeks and rather sleep
deprived while attending netdev and IETF/104.

> I'm showing you the SMBIOS values for the BIOS section.
We need to be careful in what one calls the "version"
when doing this.

> Another example: on my desktop system, 'dmidecode' shows:
> 
> 
> root@photon:/usr/home/bcran # dmidecode -t bios
> # dmidecode 3.2
> # SMBIOS entry point at 0x7949f000
> Found SMBIOS entry point in EFI, reading table from /dev/mem.
> SMBIOS 3.0 present.
  ^^^   This is the critical version, this is the
version of the SMBIOS/dmitables implemented by the bios.

Bhyve should be reporting a SMBios major 2 minor 6 (aka 2.6),
see usr.sbin/bhyve/smbiostbl.c function smbios_ep_initializer.


> Handle 0x, DMI type 0, 24 bytes
> BIOS Information
>  ??? Vendor: American Megatrends Inc.
>  ??? Version: 1701
>  ??? Release Date: 01/09/2019
>  ??? Address: 0xF
>  ??? Runtime Size: 64 kB
>  ??? ROM Size: 16 MB
>  ??? Characteristics:
>  ??? PCI is supported
>  ??? APM is supported
>  ??? BIOS is upgradeable
>  ??? BIOS shadowing is allowed
>  ??? Boot from CD is supported
>  ??? Selectable boot is supported
>  ??? BIOS ROM is socketed
>  ??? EDD is supported
>  ??? 5.25"/1.2 MB floppy services are supported (int 13h)
>  ??? 3.5"/720 kB floppy services are supported (int 13h)
>  ??? 3.5"/2.88 MB floppy services are supported (int 13h)
>  ??? Print screen service is supported (int 5h)
>  ??? 8042 keyboard services are supported (int 9h)
>  ??? Serial services are supported (int 14h)
>  ??? Printer services are supported (int 17h)
>  ??? ACPI is supported
>  ??? USB legacy is supported
>  ??? BIOS boot specification is supported
>  ??? Targeted content distribution is supported
>  ??? UEFI is supported
>  ??? BIOS Revision: 5.13
  ^^ This is an arbitrary value
that a vendor sets.  We presently report 1.0 for the legacy bios
version string, see usr.sbin/bhyyve/smbiostbl.c smbios_type1_strings[]

> -- 
> Rebecca Cran
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Updating uefi-edk2-bhyve

[Rodney W. Grimes]

> On 3/29/19 9:29 PM, Rodney W. Grimes wrote:
> >
> > That, iirc, would be the  SMBIOS version of ed2k, which yes should
> > be updated if infact the newer ed2k has a new SBIOS implementation,
> > if it is still 1.00 implementaton then this needs to be left
> > alone.
> 
> 
> Under OVMF "smbiosview -t 0" shows:

I do not know what the above is, could you elaborate for me?
What is OVMF and what is its relation to bhyve?

> 
> Vendor: EFI Development Kit II / OVMF
> 
> BiosVersion: 0.0.0
> 
> BiosReleaseDate: 02/06/2015
> 
> 
> Whereas, we have:
> 
> 
> Vendor: BHYVE
> 
> BiosVersion: 1.00
> 
> BiosReleaseDate: 03/14/2014

I have to assume this is with ed2k loaded, but I do not
know you are showing me the SMBIOS string value or some
other bios version value.  THere are compliance levels
associated with SMBIOS.

The SMBIOS versions must match what it is that is implemented,
and infact I have pending code in review that specifically
implements features as SMBIOS version 1.0, though I would
rather be implementing this in the newer SMBIOS spec (at least
2. something and preferable 3.2, but as discsussed in email
with jgb that would require a complete audit and upgrade of
our current code to be at that spec level, a none trivial,
but worthwhile effort I defered for later.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Updating uefi-edk2-bhyve

[Rodney W. Grimes]

> On 3/25/19 3:59 PM, D Scott Phillips wrote:
> >
> > Yep, makes sense to me. For either of these changes we would want to get
> > test converage on basically all functionality, so might as well take
> > both changes at once.
> 
> 
> One thing I noticed is that the SMBIOS information claims it's BHYVE 
> BIOS v1.00 released 3/14/2014. It would be nice if we could update that.

That, iirc, would be the  SMBIOS version of ed2k, which yes should
be updated if infact the newer ed2k has a new SBIOS implementation,
if it is still 1.00 implementaton then this needs to be left
alone.

> Rebecca Cran
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Rodney W. Grimes]

> > This branch is 8930 commits behind tianocore:master.
> > And depends on an external git belonging to Peter Grehan,
> 
>   No, it isn't.
> 
>   https://github.com/freebsd/uefi-edk2
> 
>   Under FreeBSD project control since Oct 16, 2015.

My appologies, the page does not clearly show who owns
the git project and I wrongly assumed it was you.

Though perhaps there is another edk2 some place
that some people seem to have copies of?

Is the source the same for the versions in
people.freebsd.org/~grehan/?


> later,
> Peter.
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Rodney W. Grimes]

> Hi all,
> 
> > Am 11.03.2019 um 19:09 schrieb Rodney W. Grimes 
> > :
> > I have no love for them either, but until we get our UEFI updated
> > it is a rather sad state of affairs.
> 
> I routinely boot FreeBSD, Ubuntu and Windows 10 with
> UEFI, so I don?t see much that I would consider broken.

All 64 bit oses, afaik we can not boot anything 32 bit
with our current uefi, which is:
This branch is 8930 commits behind tianocore:master. 
And depends on an external git belonging to Peter Grehan,
who is now gone from the project.

> 
> Missing persistence of boot variables (correct term?) is
> the only thing that I know of - what else makes it a
> sad state, currently?

Our GOP code frequently blows up vnc clients is another
issue, not sure if that is in the uefi stuff, or if we
just have bad vnc layered on top of it.  It throws recs
that are out of bounds.

The embeded PXE code is also rather dated.

> Patrick

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Rodney W. Grimes]

> On Mon, Mar 11, 2019 at 10:58:55AM -0700, Rodney W. Grimes wrote:
> > -- Start of PGP signed section.
> > > On 2019-03-11T13:08:53 -0400
> > > Shawn Webb  wrote:
> > > >
> > > > If your guest OS supports it, you could probably write two scripts that
> > > > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > > > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > > > received.
> > > > 
> > > 
> > > They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> > > one out of three of those supports it...
> > > 
> > > I suppose my other option would be to add (another) NFS mount in each
> > > guest, and have them touch a file early in the init script (and
> > > possibly touch a different file early in the shutdown script).
> > 
> > Well ICMP is in the kernel, and should be working as soon as the
> > interface is up, long before you could do anything with NFS,
> > so rather than the complexity above a simple ping would suffice.
> 
> Just a note: Windows systems disable inbound ICMP by default, but
> inbound ICMP support can be enabled post-installation.
> 
> > There is also the phase of vmm(8) startup that when you are
> > running bhyveload vs bhyve and iirc grubload vs bhyve, that
> > can be detected.  vmbhyve does so and says you are in state
> > looader when you do a vm list.
> 
> I would suggest using bhyve with UEFI. I wish a death upon bhyveload
> and grub2-bhyve.

I have no love for them either, but until we get our UEFI updated
it is a rather sad state of affairs.  If we could a) get on a modern
version of ed2k, and b) bet the CSM fixed so that we could actually
boot bios mode stuff with it and c) maybe make a port of seabios
that could be used then we would be kicking some seriuos stuff!

> Thanks,
> Shawn Webb

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Rodney W. Grimes]

> On 2019-03-11T13:27:23 -0400
> Shawn Webb  wrote:
> >
> > Both FreeBSD and Linux supports virtio_console(4). I have no idea
> > about OpenBSD, but I'm sure they'd be open to an implementation if
> > asked.
> 
> Right.
> 
> > 
> > The NFS solution would work, but it would be somewhat fragile. What
> > happens when a VM crashes? What happens when the host crashes?
> 
> At least in my case:
> 
> If the VM crashes, it'll be restarted by a process supervisor (runit,
> here).

I think ping would suffice to make that determination?

> If the host crashes, I likely have bigger problems. In any case, I
> think that's still fine because all the host would care about is if the
> guest's file was touched more recently than the last time the host
> tried to start a bhyve process for it.

You could inside the vm simply touch /tmp/foo from cron and from the host
see that the diskimage last modified time updated.  Assuming some cacheing
does not get in the way.

> I suppose I should elaborate a bit: I do have monitoring via Prometheus
> in place, but I'd like to try to stagger VM startups a little as
> starting up a lot of them in parallel on boot tends to overwhelm the
> machine slightly. Once they're all up and running in a steady state,
> things are fine. I would typically stagger the startup of
> ordinary services (inside a jail, for example) by using dependencies in
> runit - it has a facility to pause a service until a dependent service
> has been started. Unfortunately, that can't work in this case because
> once the bhyve process has been started, the host can't tell if/when
> the guest has actually fully started up. As far as runit is concerned,
> the service is up and so any dependent services should be started too.

The package vmbhyve has starggered startup in a specific ordered list
implemented.  It is all writtin in /bin/sh, so easy to adapt.

> Mark Raynsford | http://www.io7m.com
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: Detecting that a guest kernel has booted

[Rodney W. Grimes]

-- Start of PGP signed section.
> On 2019-03-11T13:08:53 -0400
> Shawn Webb  wrote:
> >
> > If your guest OS supports it, you could probably write two scripts that
> > uses virtio_console(4), one for the guest to tell the host "HELLO" and
> > one for the host to say "NICE TO SEE YOU!" once the guest's "HELLO" is
> > received.
> > 
> 
> They're a mix of FreeBSD, OpenBSD, and Debian guests. So I'm guessing
> one out of three of those supports it...
> 
> I suppose my other option would be to add (another) NFS mount in each
> guest, and have them touch a file early in the init script (and
> possibly touch a different file early in the shutdown script).

Well ICMP is in the kernel, and should be working as soon as the
interface is up, long before you could do anything with NFS,
so rather than the complexity above a simple ping would suffice.

There is also the phase of vmm(8) startup that when you are
running bhyveload vs bhyve and iirc grubload vs bhyve, that
can be detected.  vmbhyve does so and says you are in state
looader when you do a vm list.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve PCIe passthrough on AMD Epyc

[Rodney W. Grimes]

> > On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes 
> >  wrote:
> > 
> >> I'm trying to get PCIe device (video card) passthrough working to a 
> >> Windows bhyve VM, should this work now?
> > 
> > No, this should not work now.
> 
> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now?
I do not know what would be special about Epyc so can not say
for certain.

> Just trying the same thing with a simple PCIe USB card gives me:
> 
> CPU0: local APIC error 0x40
> CPU0: local APIC error 0x40
> ivhd0: Error: completion failed tail:0xda0, head:0x0.
> ivhd0: Dump all the commands:
>   [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0
>   [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7003
>   [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0
>   [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0
>   [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0
> [...snip... lots of similar stuff]
> 
> Then I get a bunch more APIC errors, and the system freezes for about 60 
> seconds before recovering.

You do not get any host dmesg errors about unaligned BAR
ranges or sizes?

Can you get the pciconf -l -b info for the device you passed, something like:
# pciconf -l -b uhci0
uhci0@pci0:0:26:0:  class=0x0c0300 card=0x20f017aa chip=0x29378086 rev=0x03 
hdr=0x00
bar   [20] = type I/O Port, range 32, base 0x1840, size 32, enabled

> -- Kevin
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve PCIe passthrough on AMD Epyc

[Rodney W. Grimes]

> I'm trying to get PCIe device (video card) passthrough working to a Windows 
> bhyve VM, should this work now?

No, this should not work now.

> I have the device blocked off in loader.conf, and added 
> hw.vmm.amdvi.enable="1".  After doing so this is how it appears in pciconf:
> 
> ppt0@pci0:97:0:0: class=0x03 card=0x27173842 chip=0x128b10de rev=0xa1 
> hdr=0x00
> bar   [10] = type Memory, range 32, base 0xe000, size 16777216, 
> disabled
> bar   [14] = type Prefetchable Memory, range 64, base 0xfcf000, size 
> 134217728, disabled
> bar   [1c] = type Prefetchable Memory, range 64, base 0xfcf800, size 
> 33554432, disabled
> bar   [24] = type I/O Port, range 32, base 0x3000, size 128, disabled
> cap 01[60] = powerspec 3  supports D0 D3  current D0
> cap 05[68] = MSI supports 1 message, 64 bit
> cap 10[78] = PCI-Express 2 legacy endpoint max data 256(256) RO NS
>  link x8(x8) speed 5.0(5.0)
> ecap 0002[100] = VC 1 max VC0
> ecap 0004[128] = Power Budgeting 1
> ecap 000b[600] = Vendor 1 ID 1
> 
> When trying to attach it to a VM with "-s 9:0,passthru,97/0/0" I get this 
> almost immediately after it starts up:
> 
> Assertion failed: (error == 0), function modify_bar_registration, file 
> /usr/src/usr.sbin/bhyve/pci_emul.c, line 504.
> 
> Also on the console I'm seeing a lot of messages like:
> 
> ivhd0: Error: completion failed tail:0x1c0, head:0x0.
> ivhd0: Dump all the commands:
>   [CMD0, off:0xff0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD1, off:0x0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD2, off:0x10] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD3, off:0x20] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD4, off:0x30] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD5, off:0x40] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD6, off:0x50] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD7, off:0x60] opcode= 0x2 0x6101 0x0 0x0
>   [CMD8, off:0x70] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD9, off:0x80] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD10, off:0x90] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD11, off:0xa0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD12, off:0xb0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD13, off:0xc0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD14, off:0xd0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD15, off:0xe0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD16, off:0xf0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD17, off:0x100] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD18, off:0x110] opcode= 0x2 0x6100 0x0 0x0
>   [CMD19, off:0x120] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD20, off:0x130] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD21, off:0x140] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD22, off:0x150] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD23, off:0x160] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD24, off:0x170] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD25, off:0x180] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD26, off:0x190] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD27, off:0x1a0] opcode= 0x1 0x304f1629 0x30 0xa5a5
>   [CMD28, off:0x1b0] opcode= 0x1 0x304f1629 0x30 0xa5a5
> 
> 
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: NVMe and Bhyve

[Rodney W. Grimes]

> On Mon, 18 Feb 2019 at 02:58, Victor Sudakov  wrote:
> 
> >
> > Or at least please tell me how I can attach another iso image (with
> > drivers) when running "vm install myhost windows.iso"
> 
> 
> Technically, you should be able to define something like:
> 
> disk1_type="ahci-cd"
> disk1_name="virtio-win-0.1.164.iso"
> disk1_dev="file"
or
disk1_dev="custom" if you wish to use a path in disk1_name,
I still do not know why this needs to be treated differently,
a path should just work :-(

> in the vm.conf file if the virtio-win-0.1.164.iso is located in
> /vm/servername/ directory.
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Multiple bhyve hosts under Linux KVM

[Rodney W. Grimes]

> Good day, 
> 
> I have found the question  on the FreeBSD wiki about running bhyve hosts 
> under Linux KVM on the bhyve's wiki:
> https://wiki.freebsd.org/bhyve#Q:_Can_I_run_multiple_bhyve_hosts_under_Linux_KVM_with_nested_VT-x_EPT.3F
>  
> 
> During the study work about virtualization I have checked it by repeating the 
> part from FreeBSD handbook: 
> https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html .
> 
> I have installed nested FreeBSD 10.3, nested Debian, and run two bhyves host 
> with nested FreeBSD 10.3 simultaneously 
> Part of that I recorded in the video: 
> https://www.youtube.com/watch?v=yMj4m0gc1_k. 
> 
> 
> In the attachments, there are my study logs. You might be interested in 
> sections "Task 2. KVM nested Bhyve" (details) and "Appendix" (some system 
> fingerprints). 
> 
> Sukhoplyuev Ilya
> 
> Content-Description: 
> I_Sukhoplyuev-LIA-Virtual-Machine-Monitors-(Hypervisors).zip

Hello,

Thank you for informing us of your work, sadly the email system
for freebsd.org has stripped your attachments from the mail so
we did not receive them.  Could you either provide a link to the
work, or if not send me a direct email with them attached
by replying to this email and I shall have the text placed in
the wiki for others to see.

Thanks,
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: The status of docker

[Rodney W. Grimes]

> On 22/01/2019 20:01, Patrick M. Hausen wrote:
> > Hi all,
> >
> >> Am 22.01.2019 um 20:48 schrieb Grzegorz Junka  >> >:
> >> 2. A docker container in a jail with no native linux kernel, docker 
> >> is running completely in FreeBSD environment
> >>
> >> 2 would be most convenient but also most difficult as all smallest 
> >> docker features would need to be ported natively to FreeBSD
> >
> > IIRC the Joyent approach was to port the Linux system call API to an
> > extent that made it possible to ?simply? run Linux Docker in zones without
> > a Linux kernel ?
> >
> 
> So option 2? That's nice, good to know. Maybe it isn't as difficult. 
> Docker is quite popular in the industry so maybe some sponsorship would 
> be possible?
> 
> BTW is Joyent is any way related to Oracle? Why would they want to 
> support zones? Is the support they implemented open sourced?

Joyent is running on Illumos which is based on OpenSolaris which
is open source.

Illumos can also run KVM and bhyve and zones all at the same time,
its rather nice in they have done some things that support this,
some of which I am working with Joyent on bringing to FreeBSD
(mainly the ability to have more than one type II hypevisor running.)

But first there is some cruft cleaning being done for both
FreeBSD and Illumos, removal of constant VM_MAXCPU so you
can run bhyve guests with as many threads as your host has.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Any plans to increase the maximum number of virtual CPUs above 16?

[Rodney W. Grimes]

> I was reading the bhyve man page and noticed that it said the maximum 
> number of virtual CPUs is limited to 16. With AMD Epyc being out with 32 
> cores and 64 threads per CPU and being able to have 2 of them in one 
> system this number seems rather low by modern standards.

So you can run 4 x 16 vcpu VM's, probably more common than
someone trying to run 1 VM of 32 or 64 vcpu.

> 
> Are there any plans to increase this number to something like 32 virtual 
> CPUs?

There is work in process to change this from a compile time
kernel constant to a per VM value set using the undocumnted,
but present value in the cpu_topology option max_cpu.

We do not want to just crank this value up as data structures 
for each VM are sized on the MAXCPU connstant, and doubling
that to 32 would be fairly wasteful for low core count VM's.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: vmrun.sh - how to pin vcpu to realcpu?

[Rodney W. Grimes]

> Hi,
> 
> As subject - how can I pin a VM that used 4x cpu (-c 4) to realcpu?
> I know that it's the -p option but am unclear on the syntax.
> 
> vmrun.sh gives the following for -p
> 
> -p: pass-through a host PCI device at bus/slot/func (e.g. 10/0/0)
> 
> but how to express this for vcpus and realcpus?

vmrun.sh and bhyve(8) are seperate commands,
only bhyve(8) supports the cpu pin option as -p,
you can not do this with the example script vmrun.sh
unless you modify it yourself and add some other
option to do this with.

To do the pinning with bhyve(8) you do something like

bhyve -p 0:0 -p 1:1 -p 2:2 -p 3:7

if I am reading the code correctly.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: vm-bhyve swap space usage

[Rodney W. Grimes]

> Hi Folks,
> 
> I've recently started using vm-bhyve-1.2.3, on FreeBSD 11.2-RELEASE-p4 and 
> noticed three processes, one for each VM, using swap as indicated by top -w.
> The processes are running the "vm" script as follows - /bin/sh 
> /usr/local/sbin/vm -tf _run {vm name}.
> Is this expected behaviour?

Have you over taxed your avaliable physical ram?  If so then yes,
this is expected behavior.  Bhyve uses "swap backed anon memory
regiones" for the memory configured to the vm.   If you do not
have that much spare free memory your system well start to swap,
including the vm processes.

You can specify a wired memory option to prevent the vm's from
using swap, but be aware that places memory pressure on other
parts of the system.  (I still stronly recommend doing this,
unless you have lots of vm's that are mostly idle and can have
slow responsiveness.)

bhyve -S option, not sure how to do that in vm-bhyve-1.2.3

Further note that if your using zfs the configuration for
it by default is to use up to 95% of your host memory
for ARC cache, this interacts very poorly with bhyve, and
I recommend reducing the max arc cache size by the amount
of memory allocated to all VM's.

sysctl vfs.zfs.arc_max


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: UEFI GOP: screen goes blank during boot after loader is finished

[Rodney W. Grimes]

> On Wed, Nov 14, 2018 at 3:49 PM Kyle Evans  wrote:
> 
> > Good, good, so the loader <-> kernel transition would appear to be
> > intact and information appears good at first blush. If you don't have
> > time to dig into this, I'll try poking at it this weekend, but it
> > looks to be either kernel problem or VNC server.
> 
> FreeBSD 11.{1,2} and various Linux distros boot fine with UEFI GOP in
> the same time, the problem is only with FreeBSD 12-ALPHA+.
> Therefore it does not look like VNC server issue.

Lets make sure this is not the invalid instruction trap (CLFLUSH)
that has already been fixed by BETA3.  Is everyone infact using
BETA3 or later as a guest?   We know of and have fixed this other
bug and I want to make sure we are not chassing a red hearing.

Thanks,
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: UEFI GOP: screen goes blank during boot after loader is finished

[Rodney W. Grimes]

> On Tue, Nov 13, 2018 at 5:15 PM Subbsd  wrote:
> >
> > Hi,
> >
> >
> >
> > On Tue, Nov 13, 2018 at 7:22 PM Rodney W. Grimes
> >  wrote:
> > >
> > > Uh oh, ok, thats not what I was thinking of then.  I do not have any
> > > bhyve capable hardware running on 12BETA at this time to test with.
> > >
> > > I wonder if we have a bad interaction between the loader and
> > > bhyve again, we have had a few of those.
> > >
> >
> > it's broken before 12-ALPHA1, my post from 14-Sep to current@
> >
> > https://lists.freebsd.org/pipermail/freebsd-current/2018-September/071206.html
> >
> > probably the lua loader doesn't understand efifb/term very well
> 
> These things are basically completely unrelated- the interpreter
> included in loader(8) doesn't have to know anything about graphics and
> this stuff is handled before and after the interpreter is invoked.
> 
> However, because loader-land is funny in a not-ha-ha kind of way, can
> you try replacing loader.efi on your guest VM with the Forth-flavored
> version to rule that out or narrow it down, please?
> 
> Thanks,

Am I miss remebering or didn't we solve a similiar problem
earlier in the release cycle?   Perhaps that was with non
GOP console only.

Either way, we need to make sure this is resolved.  Sad state is
I do not have any access until after the holiday most likely to
any hardware that I can use to work on this problem.

Please keep me in the loop though, I am reading emails each morning,
and as times allow during the day.

Thanks,
-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: UEFI GOP: screen goes blank during boot after loader is finished

[Rodney W. Grimes]

> 
> 
> 
> On November 13, 2018 at 8:36:34 AM, Rodney W. Grimes 
> (freebsd-...@pdx.rh.cn85.dnsmgr.net) wrote:
> 
> 
> Since you are using uefi and specifying a graphics console with tcp=,wait?
> your vm is waiting for a vnc connection to port 5900 to display the?
> console.?
> 
> 
> But I connect to VNC, see the loader and press Enter to boot, and *then* the 
> screen goes blank.

Uh oh, ok, thats not what I was thinking of then.  I do not have any
bhyve capable hardware running on 12BETA at this time to test with.

I wonder if we have a bad interaction between the loader and
bhyve again, we have had a few of those.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: UEFI GOP: screen goes blank during boot after loader is finished

[Rodney W. Grimes]

> I'm just getting started using bhyve (on a 13-CURRENT host from Oct 28), and 
> for the work I'm doing need to use UEFI. However, I'm finding that when 
> booting 
> FreeBSD 12.0-BETA4 the screen goes blank as FreeBSD takes over the console 
> after the loader is finished.
> 
> The command I'm running is:
> 
> sudo bhyve -A -P -c 2 -H -m 4G -s 0:0,hostbridge -s 31:0,lpc -s 2,ahci-
> cd,FreeBSD-12.0-BETA4-amd64-disc1.iso -s 
> 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait -l bootrom,/usr/local/share/uefi-
> firmware/BHYVE_UEFI.fd -u vm5
> 
> Am I missing something, or has something broken in -CURRENT or 12.0-BETA4?

Since you are using uefi and specifying a graphics console with tcp=,wait
your vm is waiting for a vnc connection to port 5900 to display the
console.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: is there a vCPU hard limit for bhyve guests?

[Rodney W. Grimes]

> Hi,
> 
> On a host with hw.ncpu=48 I tried to start a bhyve guest with -c 24 and
> it failed to start. So I tried -c 16 and it starts and runs as
> expected. Any more than -c 16 fails to run.
> 
> Is a maximum of 16 vCPU hardwired somewhere, and if so, why? Is there a
> way to overcome the limit?

man bhyve
Look at -c option, default is 1 max is 16.

There is a compile time constant VM_MAXCPU, you can raise that
to 20, then other problems come about that require more patching.

Last time I played with this it failed above 16, I suspect
some other places have had a bad derived constant using the
assumption that 16 is the max.

> thanks,
> -- 
> J.
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve uefi="csm"

[Rodney W. Grimes]

> Rodney W. Grimes wrote:
> >> Rodney W. Grimes wrote:
> >> > > Can you tell us a bit about your set up, like are you using helper 
> >> > > scripts like libvirt?
> >> > > To answer the question you asked when I run a vm with the CSM firmware 
> >> > > it works for me.
> >> >
> >> > You should always be able to boot a vm that uses
> >> > and boots with the non CSM uefi firmware with the
> >> > CSM version of the firmware, failures may start
> >> > to show up if your booting something that is
> >> > either purely legacy, or efi aware but falls
> >> > back to csm because it didnt like something.
> >>
> >> If my goal were to boot FreeDOS in bhyve, would that be ever possible?
> >
> >Today no, due to the fact we do not have a ATA/legacy
> >emulation, but rather only ahci disk emultaion, though
> >I would be wrong on that point if FreeDOS has an ahci
> >disk driver.
> 
> According to 
> http://wiki.freedos.org/wiki/index.php/Main_Page#What_makes_FreeDOS_better.3F 
> FreeDOS has "UDMA drivers for hard disks and DVD drives", is it not the 
> thing? 

No it is not the thing, UDMA was added to legacy ATA at
something like ATA-4,
Ahci does not come into until SATA existed, and iirc
thats around something like ATA-6.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve uefi="csm"

[Rodney W. Grimes]

> Rodney W. Grimes wrote:
> > > Can you tell us a bit about your set up, like are you using helper 
> > > scripts like libvirt?
> > > To answer the question you asked when I run a vm with the CSM firmware it 
> > > works for me.
> > 
> > You should always be able to boot a vm that uses
> > and boots with the non CSM uefi firmware with the
> > CSM version of the firmware, failures may start
> > to show up if your booting something that is
> > either purely legacy, or efi aware but falls
> > back to csm because it didnt like something.
> 
> If my goal were to boot FreeDOS in bhyve, would that be ever possible?

Today no, due to the fact we do not have a ATA/legacy
emulation, but rather only ahci disk emultaion, though
I would be wrong on that point if FreeDOS has an ahci
disk driver.

This goal is on my "want" list for bhyve to be able
to do.   THere is a GSOC ata driver emulation that
works some, but I do not think it can accomplish
this task at this time.

I do not know if the CSM code is up to this task
either, but it should be.

> > The older ED2K uefi that we are using and
> > some of the hatchetry that was used to craft
> > its CSM are in a poor state and could use
> > some love. 
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve uefi="csm"

[Rodney W. Grimes]

> Can you tell us a bit about your set up, like are you using helper scripts 
> like libvirt?
> To answer the question you asked when I run a vm with the CSM firmware it 
> works for me.

You should always be able to boot a vm that uses
and boots with the non CSM uefi firmware with the
CSM version of the firmware, failures may start
to show up if your booting something that is
either purely legacy, or efi aware but falls
back to csm because it didnt like something.

The older ED2K uefi that we are using and
some of the hatchetry that was used to craft
its CSM are in a poor state and could use
some love. 

> 
> 
> > On Oct 20, 2018, at 10:39 PM, Victor Sudakov  wrote:
> > 
> > Dear Colleagues,
> > 
> > Has anyone been able to boot *anything* with uefi="csm"?
> > 
> > Can you please share at least one success story?
> > 
> > -- 
> > Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
> > 2:5005/49@fidonet http://vas.tomsk.ru/
> > ___
> > freebsd-virtualization@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to 
> > "freebsd-virtualization-unsubscr...@freebsd.org"
> 
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: New bhyve user

[Rodney W. Grimes]

> >
> > ?So if one guest launch a lot of cpu hungry thread it could "starve" 
> > the others guests with less thread usgin cpu time. 
> 
> As far as what I heard here on this forum, you should NOT overcommit the 
> vCPUs.

You seemed to have heard incorrectly.  There is little to no issues
overcommiting CPU's in bhyve, I have a 2 core, 4 thread system with
6 VM's, each vm using 1 vCPU, this is a 50% overcommit and it my
base line load.

I frequently fire up 2 and 4 core VM's to do real work without
any issues.

Now, you do NOT want to end up in situation where these VM's
are over loading the actually avaliable CPU cycles, what I
am doing works becasue my base line 6 VM's only actually
consume about 1.5 cpus, so I actually do have 2.5 sitting
idle.

What does NOT work well is overcommiting Memory, that kinda
puts you in a real bad state.  I also stronly recommend 
using -S (wire memory) option, this insures you actually
can hard allocate any memory that a VM needs.

> 
> When you keep the number of vCPUs assigned to your vms lower than 
> physical CPUs (= hyperthreaded cores), you should be fine.

It is all a mater of Load, as long as your host load stays
below the number of CPU threads you actually have things
in this aspect tend to work just fine.  And if you do exceed
this everyone slows down in a fairly fair fashion.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: on bhyve statistics

[Rodney W. Grimes]

> On 8/28/18 3:37 PM, Rodney W. Grimes wrote:
> >>>> Currently, bhyve does not expose any of these statistics. All the stats
> >>> available through bhyvectl --get-stats seem to be coming from the VMM,
> >>> not from the userspace emulation.
> >>
> >>> That is correct, byhvectl is a diagnostics tool for getting
> >> information from the kernel/vmm module.
> >>
> >> bhyvectl provide stats related to processor vmx/svm from vmm.ko and is the
> >> first thing you want to run for performance regression. It will be nice to
> >> include it as part of bhyve perf tool/dashboard that you are intended to
> >> build.
> > 
> >  From conversations with Peter Grehan he expressed that bhyvectl is
> > purely a diagnostics tool that should not be depended on by any
> > other tools.
> > 
> > If you want to do similiar things you should program to the libvmmapi
> > interface, not bhyvectl.
> 
> For context, this is *not* what I'm aiming to do. While bhyvectl just 
> prints statistics exposed by the VMM through libvmmapi, I'm specifically 
> asking about instrumenting the userland part, bhyve.
> 
> Also, libvmmapi does have downstream consumers that are not 
> bhyve{,ctl,load}, e.g. grub2-bhyve, bhyve-multiboot[1], maybe even 
> xhyve[2] - for performance reasons, I'd prefer scraping metrics through 
> libvmmapi over shelling out to bhyvectl.

Yes, there are other consumers of libvmmapi, why I stressed
that it is the api to interface with vmm.ko.

It might even make since to have the userland bhyve(8)
maintain statistics in libvmmapi for exactly these reasons.

> [1] https://github.com/fubarnetes/bhyve-multiboot
> [2] https://github.com/mist64/xhyve

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: on bhyve statistics

[Rodney W. Grimes]

> On Tue, Aug 28, 2018, 9:38 PM Rodney W. Grimes <
> freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote:
> 
> > > >> Currently, bhyve does not expose any of these statistics. All the
> > stats
> > > > available through bhyvectl --get-stats seem to be coming from the VMM,
> > > > not from the userspace emulation.
> > >
> > > >That is correct, byhvectl is a diagnostics tool for getting
> > > information from the kernel/vmm module.
> > >
> > > bhyvectl provide stats related to processor vmx/svm from vmm.ko and is
> > the
> > > first thing you want to run for performance regression. It will be nice
> > to
> > > include it as part of bhyve perf tool/dashboard that you are intended to
> > > build.
> >
> > From conversations with Peter Grehan he expressed that bhyvectl is
> > purely a diagnostics tool that should not be depended on by any
> > other tools.
> >
> > If you want to do similiar things you should program to the libvmmapi
> > interface, not bhyvectl.
> >
> 
> The libvmmapi is more an internal library for usage with bhyvectl and
> bhyveload than for other purposes, of course it won't stop anybody to
> extend it to achieve other goals.

That is not my understanding at all, libvmmapi is the external and
stable/maintained interface to vmm.ko, it is not at all strickly
for use by bhyvectl or bhyveload or bhyve.
 
And as such this "api" should be very carefully maintained as
it needs to be caried forward for ever.

> > > -Anish
> > >
> > > On Mon, Aug 27, 2018 at 8:20 AM Rodney W. Grimes <
> > > freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote:
> > >
> > > > > Hi list,
> > > > >
> > > > > I'm currently looking at getting the libvirt prometheus exporter[1]
> > to
> > > > > work with libvirt+bhyve. In its current state this doesn't work
> > because
> > > > > at least one of the API calls exposed by libvirt isn't implemented by
> > > > > the libvirt bhyve driver - so I started looking at implementing it.
> > > > >
> > > > > The first API call in question is virDomainBlockStats[2], which
> > returns
> > > > > statistics (number of read and written bytes and ops, respectively).
> > > > >
> > > > > Currently, bhyve does not expose any of these statistics. All the
> > stats
> > > > > available through bhyvectl --get-stats seem to be coming from the
> > VMM,
> > > > > not from the userspace emulation.
> > > >
> > > > That is correct, byhvectl is a diagnostics tool for getting
> > > > information from the kernel/vmm module.
> > > >
> > > > > OTOH, I did see that there are *some*
> > > > > stats being collected in bhyverun.c (see struct bhyvestats {...}
> > > > > stats;). I can't see how these are exposed though -  a grep of
> > /usr/src
> > > > > turned up no other uses. Which brings me to the following questions:
> > > > >
> > > > > - are the stats in struct bhyvestats {...} stats exposed or used in
> > any
> > > > >   non-obvious way?
> > > >
> > > > Not that I am aware of.
> > > >
> > > > > - architecturally, what would be the best ways to get stats out of
> > the
> > > > >   user-space emulations? Off of the top of my head, I could think of
> > the
> > > > >   following possibilities:
> > > > >   - prometheus exporter
> > > > >   - having some socket or pipe to request them
> > > > >   - DTrace probes
> > > > >
> > > > > I wouldn't mind implementing any of the above, and so would like to
> > know
> > > > > which of these (or other options) would be the most acceptable, and
> > > > > would appreciate some guidance.
> > > >
> > > > I differ to others on what may be the best way to do this.
> > > >
> > > > > CC'ing novel@ for the libvirt side, and grehan@ for the
> > architectural
> > > > > bhyve questions.
> > > >
> > > > You should replace @grehan with @jhb,@tychon as Peter has moved on,
> > > > and John and Tycho are now the bhyve maintainers.  I was going to
> > > > add them, and remove Peter, but I see no cc: anyway, so I am sure
> > > > that they are on the virtualization list though.
> > > >
> > > > > Fabian
> > > > >
> > > > > [1] https://github.com/kumina/libvirt_exporter
> > > > > [2]
> > > >
> > https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockStats
> > > > --
> > > > Rod Grimes
> > > > rgri...@freebsd.org
> >
> > --
> > Rod Grimes
> > rgri...@freebsd.org
> > ___
> > freebsd-virtualization@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to "
> > freebsd-virtualization-unsubscr...@freebsd.org"
> >

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: on bhyve statistics

[Rodney W. Grimes]

> >> Currently, bhyve does not expose any of these statistics. All the stats
> > available through bhyvectl --get-stats seem to be coming from the VMM,
> > not from the userspace emulation.
> 
> >That is correct, byhvectl is a diagnostics tool for getting
> information from the kernel/vmm module.
> 
> bhyvectl provide stats related to processor vmx/svm from vmm.ko and is the
> first thing you want to run for performance regression. It will be nice to
> include it as part of bhyve perf tool/dashboard that you are intended to
> build.

>From conversations with Peter Grehan he expressed that bhyvectl is
purely a diagnostics tool that should not be depended on by any
other tools.

If you want to do similiar things you should program to the libvmmapi
interface, not bhyvectl.

> -Anish
> 
> On Mon, Aug 27, 2018 at 8:20 AM Rodney W. Grimes <
> freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote:
> 
> > > Hi list,
> > >
> > > I'm currently looking at getting the libvirt prometheus exporter[1] to
> > > work with libvirt+bhyve. In its current state this doesn't work because
> > > at least one of the API calls exposed by libvirt isn't implemented by
> > > the libvirt bhyve driver - so I started looking at implementing it.
> > >
> > > The first API call in question is virDomainBlockStats[2], which returns
> > > statistics (number of read and written bytes and ops, respectively).
> > >
> > > Currently, bhyve does not expose any of these statistics. All the stats
> > > available through bhyvectl --get-stats seem to be coming from the VMM,
> > > not from the userspace emulation.
> >
> > That is correct, byhvectl is a diagnostics tool for getting
> > information from the kernel/vmm module.
> >
> > > OTOH, I did see that there are *some*
> > > stats being collected in bhyverun.c (see struct bhyvestats {...}
> > > stats;). I can't see how these are exposed though -  a grep of /usr/src
> > > turned up no other uses. Which brings me to the following questions:
> > >
> > > - are the stats in struct bhyvestats {...} stats exposed or used in any
> > >   non-obvious way?
> >
> > Not that I am aware of.
> >
> > > - architecturally, what would be the best ways to get stats out of the
> > >   user-space emulations? Off of the top of my head, I could think of the
> > >   following possibilities:
> > >   - prometheus exporter
> > >   - having some socket or pipe to request them
> > >   - DTrace probes
> > >
> > > I wouldn't mind implementing any of the above, and so would like to know
> > > which of these (or other options) would be the most acceptable, and
> > > would appreciate some guidance.
> >
> > I differ to others on what may be the best way to do this.
> >
> > > CC'ing novel@ for the libvirt side, and grehan@ for the architectural
> > > bhyve questions.
> >
> > You should replace @grehan with @jhb,@tychon as Peter has moved on,
> > and John and Tycho are now the bhyve maintainers.  I was going to
> > add them, and remove Peter, but I see no cc: anyway, so I am sure
> > that they are on the virtualization list though.
> >
> > > Fabian
> > >
> > > [1] https://github.com/kumina/libvirt_exporter
> > > [2]
> > https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockStats
> > --
> > Rod Grimes
> > rgri...@freebsd.org

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: on bhyve statistics

[Rodney W. Grimes]

> Hi list,
> 
> I'm currently looking at getting the libvirt prometheus exporter[1] to
> work with libvirt+bhyve. In its current state this doesn't work because
> at least one of the API calls exposed by libvirt isn't implemented by
> the libvirt bhyve driver - so I started looking at implementing it.
> 
> The first API call in question is virDomainBlockStats[2], which returns
> statistics (number of read and written bytes and ops, respectively).
> 
> Currently, bhyve does not expose any of these statistics. All the stats
> available through bhyvectl --get-stats seem to be coming from the VMM,
> not from the userspace emulation.

That is correct, byhvectl is a diagnostics tool for getting
information from the kernel/vmm module.

> OTOH, I did see that there are *some*
> stats being collected in bhyverun.c (see struct bhyvestats {...}
> stats;). I can't see how these are exposed though -  a grep of /usr/src
> turned up no other uses. Which brings me to the following questions:
> 
> - are the stats in struct bhyvestats {...} stats exposed or used in any
>   non-obvious way?

Not that I am aware of.

> - architecturally, what would be the best ways to get stats out of the
>   user-space emulations? Off of the top of my head, I could think of the
>   following possibilities:
>   - prometheus exporter
>   - having some socket or pipe to request them
>   - DTrace probes
> 
> I wouldn't mind implementing any of the above, and so would like to know
> which of these (or other options) would be the most acceptable, and
> would appreciate some guidance.

I differ to others on what may be the best way to do this.

> CC'ing novel@ for the libvirt side, and grehan@ for the architectural
> bhyve questions.

You should replace @grehan with @jhb,@tychon as Peter has moved on,
and John and Tycho are now the bhyve maintainers.  I was going to
add them, and remove Peter, but I see no cc: anyway, so I am sure
that they are on the virtualization list though.

> Fabian
> 
> [1] https://github.com/kumina/libvirt_exporter
> [2] https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockStats
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: access usb device from bhyve vm

[Rodney W. Grimes]

> I asked this on usb@, but might get a better bhyve perspective on this
> list...
> 
> Does anyone have thought about what it would take to support access to
> individual usb devices "natively" from a bhyve vm?

The best though on this, and note I have not done this with bhyve,
but have done it with ESXi, is to get yourself a PCI USB card,
and pass the card in as a PCI device.

Note that for bhyve this requires a careful selection of
that PCI card, as the BAR's have to be page sized and
paged aligned to not have some issues getting it passed
in due to the fact that the IOMMU only works on pages
aligned and sized stuff.

I have asked about what magic would it take to make our
USB stack allow this and it is "non trivial".

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Checking bhyve supported features (sysctls)

[Rodney W. Grimes]

> From: owner-freebsd-virtualizat...@freebsd.org 
>  On Behalf Of Rodney W. Grimes
> Sent: 16 August 2018 18:31
> To: Allan Jude 
> Cc: Matt Churchyard ; 
> freebsd-virtualization@freebsd.org
> Subject: Re: Checking bhyve supported features (sysctls)
> 
> > On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" 
> >  wrote:
> > >> 
> > >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> > >> 
> > >> > > Hello,
> > >> > > 
> > >> > > I'm looking for better ways to check for  bhyve support /
> > >available
> > >> > > features without trying to scan through dmesg output.
> > >> > 
> > >> > >Yes, it would be very good to remove that, as it usually tries 
> > >> > >to grep a non-existent file /var/run/dmesg.boot that is not 
> > >> > >created until after vm_bhyve has been called from
> > >/usr/local/etc/rc.d
> > >> > >when you have things set to autostartup >in /etc/rc.conf
> > >> > 
> > >> > 
> > >> > > 
> > >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> > >soon
> > >> > > as the vmm module is loaded
> > >> > > 
> > >> > > hw.vmm.vmx.initialized: 1
> > >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> > >> > > 
> > >> > > Will these be available on both Intel & AMD processors as a way 
> > >> > > to determine if the module has loaded successfully and can run
> > >guests?
> > >> > > 
> > >> > > I also see the below sysctl related to iommu.
> > >> > > 
> > >> > > hw.vmm.iommu.initialized
> > >> > > 
> > >> > > Again, will this be set to 1 as soon as the module is loaded if 
> > >> > > iommu is supported, or only when it is used?
> > >> > > There also seems to be a vmm.amdvi.enable sysctl.
> > >> > > Would both these need checking or is vmm.iommu enough to 
> > >> > > determine support on any processor.
> > >> > 
> > >> > >Probalby the safest way for a shell script to decide if bhyve is 
> > >> > >up and running is to stat /dev/vmm, if that exists then the
> > >modules
> > >> > >have loaded and initialized and bhyve should be ready to process
> > >guests.
> > >> > 
> > >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> > >> 
> > >> I'll investigate that, I was pretty sure that you should get this 
> > >> as soon as the vmm.ko module is finished initialzing, but you might 
> > >> be right in that it takes a first vm to cause its creation.
> > >> Confirmed, /dev/vmm does not exist until the first vm is created.
> > >> 
> > >> > 
> > >> > >sysctl's mentiond above would be a poor way to make this
> > >determination.
> > >> > 
> > >> > It would be nice if sysctls were better documented.
> > >> 
> > >> Agreed.
> > >> 
> > >> > If vmx.initialized is set once vmm has successfully loaded, I 
> > >> > can't
> > >see a better way of checking for bhyve support (assuming it's not 
> > >Intel specific). This entry definitely exists and is set to 0 if you 
> > >load the module on a non-supported system, and set to 1 as soon as 
> > >vmm loads on my Intel test system.
> > >> 
> > >> Given its undocumented status you would be relying on an 
> > >> undocumented feature that could change in either name or behavior, 
> > >> and that is not desirable.
> > >> 
> > >> Let me see if I can come up with something else.
> > >
> > >I looked at the code for bhyvectl, bhyveload and byhve.  They do not 
> > >actually try to decide if vmm is supported or not, they simply 
> > >process the error from a vm_create() or vm_open() call and exit with 
> > >an error code if they can not handle it (some of the code can handle 
> > >a vm_create failure if infact we are trying to create a vm that 
> > >already exists).
> > >
> > >If you want to maintain full compatibility a similiar stratergy may 
> > >be in order.
> > >
> > >Why is it that vm-bhyve specifically needs to know if the kernel has 
> > >v

Re: Checking bhyve supported features (sysctls)

[Rodney W. Grimes]

> On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" 
>  wrote:
> >> 
> >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> >> 
> >> > > Hello,
> >> > > 
> >> > > I'm looking for better ways to check for  bhyve support /
> >available
> >> > > features without trying to scan through dmesg output.
> >> > 
> >> > >Yes, it would be very good to remove that, as it usually tries
> >> > >to grep a non-existent file /var/run/dmesg.boot that is not
> >> > >created until after vm_bhyve has been called from
> >/usr/local/etc/rc.d
> >> > >when you have things set to autostartup >in /etc/rc.conf
> >> > 
> >> > 
> >> > > 
> >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> >soon 
> >> > > as the vmm module is loaded
> >> > > 
> >> > > hw.vmm.vmx.initialized: 1
> >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> >> > > 
> >> > > Will these be available on both Intel & AMD processors as a way
> >> > > to determine if the module has loaded successfully and can run
> >guests?
> >> > > 
> >> > > I also see the below sysctl related to iommu.
> >> > > 
> >> > > hw.vmm.iommu.initialized
> >> > > 
> >> > > Again, will this be set to 1 as soon as the module is loaded if
> >> > > iommu is supported, or only when it is used?
> >> > > There also seems to be a vmm.amdvi.enable sysctl.
> >> > > Would both these need checking or is vmm.iommu enough to
> >> > > determine support on any processor.
> >> > 
> >> > >Probalby the safest way for a shell script to decide if bhyve is
> >> > >up and running is to stat /dev/vmm, if that exists then the
> >modules
> >> > >have loaded and initialized and bhyve should be ready to process
> >guests.
> >> > 
> >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> >> 
> >> I'll investigate that, I was pretty sure that you should get this
> >> as soon as the vmm.ko module is finished initialzing, but you might
> >> be right in that it takes a first vm to cause its creation.
> >> Confirmed, /dev/vmm does not exist until the first vm
> >> is created.
> >> 
> >> > 
> >> > >sysctl's mentiond above would be a poor way to make this
> >determination.
> >> > 
> >> > It would be nice if sysctls were better documented.
> >> 
> >> Agreed.
> >> 
> >> > If vmx.initialized is set once vmm has successfully loaded, I can't
> >see a better way of checking for bhyve support (assuming it's not Intel
> >specific). This entry definitely exists and is set to 0 if you load the
> >module on a non-supported system, and set to 1 as soon as vmm loads on
> >my Intel test system.
> >> 
> >> Given its undocumented status you would be relying on an
> >> undocumented feature that could change in either name or
> >> behavior, and that is not desirable.
> >> 
> >> Let me see if I can come up with something else.
> >
> >I looked at the code for bhyvectl, bhyveload and
> >byhve.  They do not actually try to decide if vmm
> >is supported or not, they simply process the error
> >from a vm_create() or vm_open() call and exit
> >with an error code if they can not handle it
> >(some of the code can handle a vm_create failure
> >if infact we are trying to create a vm that
> >already exists).
> >
> >If you want to maintain full compatibility a similiar
> >stratergy may be in order.
> >
> >Why is it that vm-bhyve specifically needs to know
> >if the kernel has vmm support or not?
> >Cant it just be written to handle the errors returned
> >if the supported functions do not exist?
> 
> I think the question vm-bhyve wants to answer is: does
> the CPU have the required features to run a multicore VM.

Why does it need to know that?  If it tries to start a multicore/thread
VM and the system can not support it an error is returned and the bhyve
command fails.

Actually determining that specific issue is non-trivial iirc as
some vmm supported CPU's can only run a single VM with a single
thread in that VM (early core cpu's).

> 
> These or similar sysctls do seem to be the correct way
> to communicate that support.

I do not believe any

Re: Checking bhyve supported features (sysctls)

[Rodney W. Grimes]

> 
> Text manually wrapped to 80, any broken quoting is my fault - rwg
> 
> > > Hello,
> > > 
> > > I'm looking for better ways to check for  bhyve support / available
> > > features without trying to scan through dmesg output.
> > 
> > >Yes, it would be very good to remove that, as it usually tries
> > >to grep a non-existent file /var/run/dmesg.boot that is not
> > >created until after vm_bhyve has been called from /usr/local/etc/rc.d
> > >when you have things set to autostartup >in /etc/rc.conf
> > 
> > 
> > > 
> > > I notice that the following 2 sysctl's appear to be set to 1 as soon 
> > > as the vmm module is loaded
> > > 
> > > hw.vmm.vmx.initialized: 1
> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> > > 
> > > Will these be available on both Intel & AMD processors as a way
> > > to determine if the module has loaded successfully and can run guests?
> > > 
> > > I also see the below sysctl related to iommu.
> > > 
> > > hw.vmm.iommu.initialized
> > > 
> > > Again, will this be set to 1 as soon as the module is loaded if
> > > iommu is supported, or only when it is used?
> > > There also seems to be a vmm.amdvi.enable sysctl.
> > > Would both these need checking or is vmm.iommu enough to
> > > determine support on any processor.
> > 
> > >Probalby the safest way for a shell script to decide if bhyve is
> > >up and running is to stat /dev/vmm, if that exists then the modules
> > >have loaded and initialized and bhyve should be ready to process guests.
> > 
> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> 
> I'll investigate that, I was pretty sure that you should get this
> as soon as the vmm.ko module is finished initialzing, but you might
> be right in that it takes a first vm to cause its creation.
> Confirmed, /dev/vmm does not exist until the first vm
> is created.
> 
> > 
> > >sysctl's mentiond above would be a poor way to make this determination.
> > 
> > It would be nice if sysctls were better documented.
> 
> Agreed.
> 
> > If vmx.initialized is set once vmm has successfully loaded, I can't see a 
> > better way of checking for bhyve support (assuming it's not Intel 
> > specific). This entry definitely exists and is set to 0 if you load the 
> > module on a non-supported system, and set to 1 as soon as vmm loads on my 
> > Intel test system.
> 
> Given its undocumented status you would be relying on an
> undocumented feature that could change in either name or
> behavior, and that is not desirable.
> 
> Let me see if I can come up with something else.

I looked at the code for bhyvectl, bhyveload and
byhve.  They do not actually try to decide if vmm
is supported or not, they simply process the error
from a vm_create() or vm_open() call and exit
with an error code if they can not handle it
(some of the code can handle a vm_create failure
if infact we are trying to create a vm that
already exists).

If you want to maintain full compatibility a similiar
stratergy may be in order.

Why is it that vm-bhyve specifically needs to know
if the kernel has vmm support or not?
Cant it just be written to handle the errors returned
if the supported functions do not exist?


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Checking bhyve supported features (sysctls)

[Rodney W. Grimes]

Text manually wrapped to 80, any broken quoting is my fault - rwg

> > Hello,
> > 
> > I'm looking for better ways to check for  bhyve support / available
> > features without trying to scan through dmesg output.
> 
> >Yes, it would be very good to remove that, as it usually tries
> >to grep a non-existent file /var/run/dmesg.boot that is not
> >created until after vm_bhyve has been called from /usr/local/etc/rc.d
> >when you have things set to autostartup >in /etc/rc.conf
> 
> 
> > 
> > I notice that the following 2 sysctl's appear to be set to 1 as soon 
> > as the vmm module is loaded
> > 
> > hw.vmm.vmx.initialized: 1
> > hw.vmm.vmx.cap.unrestricted_guest: 1
> > 
> > Will these be available on both Intel & AMD processors as a way
> > to determine if the module has loaded successfully and can run guests?
> > 
> > I also see the below sysctl related to iommu.
> > 
> > hw.vmm.iommu.initialized
> > 
> > Again, will this be set to 1 as soon as the module is loaded if
> > iommu is supported, or only when it is used?
> > There also seems to be a vmm.amdvi.enable sysctl.
> > Would both these need checking or is vmm.iommu enough to
> > determine support on any processor.
> 
> >Probalby the safest way for a shell script to decide if bhyve is
> >up and running is to stat /dev/vmm, if that exists then the modules
> >have loaded and initialized and bhyve should be ready to process guests.
> 
> Hmm, I don't get /dev/vmm unless I actually have running guests.

I'll investigate that, I was pretty sure that you should get this
as soon as the vmm.ko module is finished initialzing, but you might
be right in that it takes a first vm to cause its creation.
Confirmed, /dev/vmm does not exist until the first vm
is created.

> 
> >sysctl's mentiond above would be a poor way to make this determination.
> 
> It would be nice if sysctls were better documented.

Agreed.

> If vmx.initialized is set once vmm has successfully loaded, I can't see a 
> better way of checking for bhyve support (assuming it's not Intel specific). 
> This entry definitely exists and is set to 0 if you load the module on a 
> non-supported system, and set to 1 as soon as vmm loads on my Intel test 
> system.

Given its undocumented status you would be relying on an
undocumented feature that could change in either name or
behavior, and that is not desirable.

Let me see if I can come up with something else.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Checking bhyve supported features (sysctls)

[Rodney W. Grimes]

> Hello,
> 
> I'm looking for better ways to check for bhyve support / available features 
> without trying to scan through dmesg output.

Yes, it would be very good to remove that,
as it usually tries to grep a non-existent file /var/run/dmesg.boot
that is not created until after vm_bhyve has been
called from /usr/local/etc/rc.d when you have things
set to autostartup in /etc/rc.conf


> 
> I notice that the following 2 sysctl's appear to be set to 1 as soon as the 
> vmm module is loaded
> 
> hw.vmm.vmx.initialized: 1
> hw.vmm.vmx.cap.unrestricted_guest: 1
> 
> Will these be available on both Intel & AMD processors as a way to determine 
> if the module has loaded successfully and can run guests?
> 
> I also see the below sysctl related to iommu.
> 
> hw.vmm.iommu.initialized
> 
> Again, will this be set to 1 as soon as the module is loaded if iommu is 
> supported, or only when it is used?
> There also seems to be a vmm.amdvi.enable sysctl. Would both these need 
> checking or is vmm.iommu enough to determine support on any processor.

Probalby the safest way for a shell script to decide if
bhyve is up and running is to stat /dev/vmm, if that
exists then the modules have loaded and initialized
and bhyve should be ready to process guests.

sysctl's mentiond above would be a poor way to make
this determination.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Curent Centos 7 and bhyve

[Rodney W. Grimes]

> > Rodney W. Grimes wrote:
> 
> [dd]
> 
> > > > > > 
> > > > > > Though it has a lot of features, it also has some short 
> > > > > > comings, like you can not spec a vm to be wired in memory, 
> > > > > > which IMHO is the only way to insure consistent VM performance.
> > > > > 
> > > > > Well, we have "bhyve_options" configuration option in the vm 
> > > > > config, why not put "-S" there, is that what you mean by wiring 
> > > > > the vm in memory?
> > > > 
> > > > I believe that fails as that only adds the -S to bhyve, and you 
> > > > must specify it both on bhyveload and bhyve for it to work.
> > > 
> > > I think it is totally doable becase vm-bhyve is nothing but a suit 
> > > of scripts. A PR with a feature request would be appropriate.
> > 
> > I made several attempts to contact the author at the email address 
> > provided at the git hub while making other bhyve changes to try and 
> > coordinate with him.  I got no response after 3 attempts,
> > so have stopped trying to contact them.   (This was while I was
> > adding the -c cpu topology modifications.)
> 
> > You can add yourself to
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230580
> > maybe something useful comes out of it.
> 
> I've already commented on the bug, although I'll reply here as well.
> If "-S" is found in bhyve_options it does currently affect both commands. I 
> have decided that a specific wired_memory option is useful though and will 
> add this to the next release.
> 
> The name limit has been increased to 32 since v1.2

This is better, but still an artificial limit, the implementation
of bhyve allows this string to be any valid filename, the scripts
should be designed to allow for that as the valid limit.

I modified the script so that the vm name is the last column,
and removed the length check all togeather, this allows for
the string to be what ever length and not mess with column
widths.

root@x230a:# vm list
DATASTORE   LOADER  CPUMEMORYVNC  AUTOSTART
STATENAME
default bhyveload   1  128M  -No   
Stopped  fb-bld-10-amd64
default bhyveload   4  2048M -No   
Stopped  fb-bld-11-amd64
default bhyveload   4  1024M -No   
Stopped  fb-bld-11-i386
default bhyveload   1  128M  -No   
Stopped  fb-bld-11.0-p1-amd64
default bhyveload   1  128M  -No   
Stopped  fb-bld-11.0-p1-i386
default bhyveload   4  512M  -No   
Stopped  fb-bld-11.1-amd64
default bhyveload   4  512M  -No   
Stopped  fb-bld-11.1-i386


> I didn't realize the changes for cpu topology had actually made it
> into head, although I don't believe it's actually in a release yet?

Yes, they are in head, the MFC has been delayed for other reasons.
It is in the 12.0-ALPHA1 snapshot, and many before that.  

> I will plan to support configuration and display of these
> before 12 release.

Thanks.  I think mostly just extract the NCPU's from the
topology string.  The code actually works now, but due
to fixed column width assumptions the output looks bad.

> 
> Matt
> 
> > --
> > Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
> > AS43859
> > ___
> > freebsd-virtualization@freebsd.org mailing list 
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to 
> > "freebsd-virtualization-unsubscr...@freebsd.org"
> 
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Curent Centos 7 and bhyve

[Rodney W. Grimes]

> Rodney W. Grimes wrote:
> 
> [dd]
> 
> > > > > > 
> > > > > > Though it has a lot of features, it also has some short comings,
> > > > > > like you can not spec a vm to be wired in memory, which IMHO is
> > > > > > the only way to insure consistent VM performance.
> > > > > 
> > > > > Well, we have "bhyve_options" configuration option in the vm config,
> > > > > why not put "-S" there, is that what you mean by wiring the vm in
> > > > > memory?
> > > > 
> > > > I believe that fails as that only adds the -S to bhyve, and
> > > > you must specify it both on bhyveload and bhyve for it to
> > > > work.
> > > 
> > > I think it is totally doable becase vm-bhyve is nothing but a suit of
> > > scripts. A PR with a feature request would be appropriate.
> > 
> > I made several attempts to contact the author at the email address
> > provided at the git hub while making other bhyve changes to try
> > and coordinate with him.  I got no response after 3 attempts,
> > so have stopped trying to contact them.   (This was while I was
> > adding the -c cpu topology modifications.)
> 
> You can add yourself to
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230580
> maybe something useful comes out of it.

I added virtualization@ to the PR, though it looks as if
wired_memory= has already been added as an option, which
is what I did locally.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Curent Centos 7 and bhyve

[Rodney W. Grimes]

> Rodney W. Grimes wrote:
> > > Rodney W. Grimes wrote:
> > > > > > > > > > Are there issues with Current CEntos and bhyve?
> > > > > > > > > 
> > > > > > > > > Sure there are, please look at
> > > > > > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230453
> > > > > > > > 
> > > > > > > > Booting in UEFI mode works.
> > > > > > > 
> > > > > > > This means we need an update to 
> > > > > > > /usr/local/share/examples/vm-bhyve/centos7.conf ?
> > > > > > > It says 'loader="grub"' for the present.
> > > > > > > 
> > > > > > > Do you have a vm config to boot centos7 in UEFI mode you could 
> > > > > > > share?
> > > > > > 
> > > > > > I just use /usr/share/examples/bhyve/vmrun.sh. I don't use any
> > > > > > third-party utility to manage bhyve VMs. vmrun.sh is pretty
> > > > > > straight-forward.
> > > > > 
> > > > > Thanks for replying. However, I highly recommend vm-bhyve, maybe you
> > > > > should give it a try. You will love the ease of VM creation and
> > > > > provisioning, network management, ZFS integration (VM snapshots and
> > > > > cloning), console and datastore management etc.
> > > > 
> > > > Though it has a lot of features, it also has some short comings,
> > > > like you can not spec a vm to be wired in memory, which IMHO is
> > > > the only way to insure consistent VM performance.
> > > 
> > > Well, we have "bhyve_options" configuration option in the vm config,
> > > why not put "-S" there, is that what you mean by wiring the vm in
> > > memory?
> > 
> > I believe that fails as that only adds the -S to bhyve, and
> > you must specify it both on bhyveload and bhyve for it to
> > work.
> 
> I think it is totally doable becase vm-bhyve is nothing but a suit of
> scripts. A PR with a feature request would be appropriate.

I made several attempts to contact the author at the email address
provided at the git hub while making other bhyve changes to try
and coordinate with him.  I got no response after 3 attempts,
so have stopped trying to contact them.   (This was while I was
adding the -c cpu topology modifications.)

> 
> What about VM that don't use bhyveload, but some other kind of loader
> like grub2-bhyve?

I am not sure how vm-bhyve deals with that as I have none of
those type VM's.
> 
> > > > 
> > > > Its artificial restriction of 16 character VM names is also
> > > > a fair bit annoying.
> > > 
> > > Maybe.
> > 
> > Maybe?  No, factually.  I migrated a number of ESXi VM's and
> > had to patch vm-bhyve to not have this restriction, so it is
> > annoying.
> 
> Did you send your patches upstream? 

Could not even get an email ack from upstream.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Curent Centos 7 and bhyve

[Rodney W. Grimes]

> Rodney W. Grimes wrote:
> > > > > > > > Are there issues with Current CEntos and bhyve?
> > > > > > > 
> > > > > > > Sure there are, please look at
> > > > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230453
> > > > > > 
> > > > > > Booting in UEFI mode works.
> > > > > 
> > > > > This means we need an update to 
> > > > > /usr/local/share/examples/vm-bhyve/centos7.conf ?
> > > > > It says 'loader="grub"' for the present.
> > > > > 
> > > > > Do you have a vm config to boot centos7 in UEFI mode you could share?
> > > > 
> > > > I just use /usr/share/examples/bhyve/vmrun.sh. I don't use any
> > > > third-party utility to manage bhyve VMs. vmrun.sh is pretty
> > > > straight-forward.
> > > 
> > > Thanks for replying. However, I highly recommend vm-bhyve, maybe you
> > > should give it a try. You will love the ease of VM creation and
> > > provisioning, network management, ZFS integration (VM snapshots and
> > > cloning), console and datastore management etc.
> > 
> > Though it has a lot of features, it also has some short comings,
> > like you can not spec a vm to be wired in memory, which IMHO is
> > the only way to insure consistent VM performance.
> 
> Well, we have "bhyve_options" configuration option in the vm config,
> why not put "-S" there, is that what you mean by wiring the vm in
> memory?

I believe that fails as that only adds the -S to bhyve, and
you must specify it both on bhyveload and bhyve for it to
work.

> > 
> > Its artificial restriction of 16 character VM names is also
> > a fair bit annoying.
> 
> Maybe.

Maybe?  No, factually.  I migrated a number of ESXi VM's and
had to patch vm-bhyve to not have this restriction, so it is
annoying.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Curent Centos 7 and bhyve

[Rodney W. Grimes]

> Shawn Webb wrote:
> > > > > > Are there issues with Current CEntos and bhyve?
> > > > > 
> > > > > Sure there are, please look at
> > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230453
> > > > 
> > > > Booting in UEFI mode works.
> > > 
> > > This means we need an update to 
> > > /usr/local/share/examples/vm-bhyve/centos7.conf ?
> > > It says 'loader="grub"' for the present.
> > > 
> > > Do you have a vm config to boot centos7 in UEFI mode you could share?
> > 
> > I just use /usr/share/examples/bhyve/vmrun.sh. I don't use any
> > third-party utility to manage bhyve VMs. vmrun.sh is pretty
> > straight-forward.
> 
> Thanks for replying. However, I highly recommend vm-bhyve, maybe you
> should give it a try. You will love the ease of VM creation and
> provisioning, network management, ZFS integration (VM snapshots and
> cloning), console and datastore management etc.

Though it has a lot of features, it also has some short comings,
like you can not spec a vm to be wired in memory, which IMHO is
the only way to insure consistent VM performance.

Its artificial restriction of 16 character VM names is also
a fair bit annoying.

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Overcommitting CPUs with BHyve?

[Rodney W. Grimes]

> What are people's experiences with overcommitting CPUs in BHyve?  I have an
> 8-core machine that often runs VMs totalling up to 5 allocated CPUs without
> problems.  But today I got greedy.  I assigned 8 cores to one VM for a big
> build job.  Obviously, some of those were shared with the host.  I also
> assigned it 8GB of RAM (out of 16 total).  Build performance fell through
> the floor, even though the host was idle.  Eventually I killed the build
> and restarted it with a more modest 2 make jobs (but the VM still had 8
> cores).  Performance improved.  But eventually the system seemed to be
> mostly hung, while I had a build job running on the host as well as in the
> VM.  I killed both build jobs, which resolved the hung processes.  Then I
> restarted the host's build alone, and my system completely hung, with
> top(1) indicating that many processes were in the pfault state.
> 
> So my questions are:
> 1) Is it a known problem to overcommit CPUs with BHyve?
> 2) Could this be related to the pfault hang, even though the guest was idle
> at the time?

I on occasion do over commit vCPU's in bhyve, but I do so
with a few specific conditions:

1) I count CPU's as real Cores, not Hyperthread cores, I do not
   expect hyperthreading to work well in over commit.

2) I always wire my VM's memory, I NEVER overcommit memory, that
   just leads to bad and ugly.  (-S option to bhyveload and bhyve)
   This totally takes ARC issues out of the picture, but you may
   not be able to start your VM's if you dont decrease the ARC.

3) Watch out for host side disk drive IOP saturation, you can easily
   stahly your guests if your trying to do to much I/O, they usually
   recover from this on there own, though it can make things go
   pretty slow for a time.   Firing off 16 VM's doing "nightly" on
   a single spindle host is a sure way to have some very long runs.

My work load runs from an always running 6 vCPU light load,
to an occasion guest running make -j4 buildworlds (total of 10 vCPU
load).  My host has 4 cores, 8 threads.  The 10vCPU load usually
drives the host to a load average of 5.

The 6 vCPU always present light load very rarely drives the
load above 1.

I think the secret sauce is wired memory :-)

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve and freebsd memstick installer

[Rodney W. Grimes]

> Hello,
> 
> context: freebsd-12-current server, amd64
> 
> I usually install a freebsd guest like this:
> 
> sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 4096M -t tap0 -d guest.img 
> -i -I FreeBSD-installation-dvd1.iso guestname
> 
> I only have memstick.img - how do I either:
> 
> 1. convert the memstick to dvd1.iso
> 2. make bhyve use the memstick installation iso (I don't think it can do 
> this as the option to do so isn't in vmrun.sh)

Untested, but since memstick is actually a "disk" image and not
a "cdrom" image you should be able to do something like
vmrun.sh -c 2 -m 4096M -t tap0 -d memstick.img -d guest.img guestname

Your memstick should show up as ada0,
and your guest.img should be ada1.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: freebsd bhyve instance does not show kernel messages after boot screen

[Rodney W. Grimes]

> On 14/06/2018 23:26, David P. Discher wrote:
> > Try in /boot/loader.conf of the VM :
> > 
> > console=userboot
> > 
> > or after beastie drop to loader OK promot ?and try :
> > 
> > set console=userboot
> > 
> > I think 11.x should fall back to userboot in?bhyve if vidconsole of 
> > comconsole were set.
> > 
> > (This is assuming non-EFI booting - using bhyveloader ).
> 
> Hi,
> 
> Thanks for trying to help. I tried it both ways - in loader.conf and 
> from the loader prompt but unfortunately it didn't work. For 
> clarification, here's the exact way I start the VM:
> 
> 1. $ screen -S fbsd11-vm
> 2. $ sudo su -
> 3. # (cd to where vm is)
> 4. sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 8192M -t tap3 -d 
> fbsd-guest.img fbsd-guest
> 
> If I transfer the VM to a freebsd-11 host (r333874) it also happens. So 
> the behaviour is a property of the VM. Unless of course the issue is 
> with bhyve itself on both -stable and -current.
> 
> A possible clue (but I'm really out of my depth here) is ISTR something 
> like this happening if say the VM is called say fbsd-guest and is then 
> subsequently launched with some other name like fbsd11guest.
> 
> Any ideas?

With the VM shutdown look in /dev/vmm for the same name as the VM,
if you see it there, make sure you do not have a running instnace
of it, then do:
bhyvectl --destroy --name=fbsd11-vm

You may have remanants of a prior/crashed VM hanging around
causing you issues.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: ARMv8 development board with GICv3

[Rodney W. Grimes]

> On 2018-06-13 09:06, Marcelo Araujo wrote:
> > 2018-06-13 14:54 GMT+08:00 Alexandru Elisei 
> > :
> > 
> >> Hello,
> >> 
> >> I have been working on porting bhyve to ARMv8 and the hypervisor is
> >> able to successfully boot a FreeBSD virtual machine on the Foundation
> >> Emulator provided by ARM.
> >> 
> >> I plan to submit the project for review, but before that I need to
> >> validate the hypervisor on a hardware platform.
> >> 
> >> Can anyone be so kind as to recommend a development board for testing?
> >> The board needs to have an ARMv8 CPU with virtualization extensions
> >> implemented (Exception Level 2 needs to be available) and a GIC
> >> version 3 compliant interrupt controller.
> >> 
> >> Thank you,
> >> Alexandru Elisei
> > 
> > Hello Alexandru,
> > 
> > Excited to see your work!
> > 
> > I have Cc manu@ as I know he works a lot with embedded devices and 
> > probably
> > he can give you some suggestions.
> > I'm sure he is on freebsd-arm@ mailing list, but even though I'm Cc'ing 
> > him.
> > 
> > Best,
> > --
> 
>   A quick grep in the linux dts directory for arm64 give me a few SoC 
> where we boot on it :
> 
>   Thunderx and Marvell 37XX.
> 
>   That being said I think you better try with a gic-v2 board as I adviced 
> in AsiaBSDCon this year.

Emmanuel,

Can you tell me what GIC is in the Samsung Chromebook SNOW?
https://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices/samsung-arm-chromebook

Thanks,
Rod

> Emmanuel Vadot  

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: ARMv8 development board with GICv3

[Rodney W. Grimes]

> Hello,
> 
> I have been working on porting bhyve to ARMv8 and the hypervisor is
> able to successfully boot a FreeBSD virtual machine on the Foundation
> Emulator provided by ARM.
> 
> I plan to submit the project for review, but before that I need to
> validate the hypervisor on a hardware platform.
> 
> Can anyone be so kind as to recommend a development board for testing?
> The board needs to have an ARMv8 CPU with virtualization extensions
> implemented (Exception Level 2 needs to be available) and a GIC
> version 3 compliant interrupt controller.

I believe the Chromebook SNOW meets the criteria, not exactly a
"board" but I have one on loan from Michael Dexter that was originally
purhased to support ARMv8 bhyve work.

Not sure that it has a version 3 GIC either.  Actually I am thinking
that you have to do the software GIC on this platform.

I have booted various versions of FreeBSD on it, the most annoying
problem is that the built in keyboard goes into continuous autorepeat
as soon as you try to login, but a USB attached keyboard works fine.

> Thank you,
> Alexandru Elisei

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve networking

[Rodney W. Grimes]

thernet 10Gbase-T 
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=63<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> inet 127.0.0.1 netmask 0xff00
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> groups: lo
> $ ping 172.16.92.11
> PING 172.16.92.11 (172.16.92.11): 56 data bytes
> 64 bytes from 172.16.92.11: icmp_seq=0 ttl=64 time=0.416 ms
> 64 bytes from 172.16.92.11: icmp_seq=1 ttl=64 time=0.371 ms
> 64 bytes from 172.16.92.11: icmp_seq=2 ttl=64 time=0.369 ms
> --- 172.16.92.11 ping statistics ---
> 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip 
> min/avg/max/stddev = 0.369/0.385/0.416/0.022 ms
> 
> $ telnet bbc.co.uk 443
> Trying 151.101.192.81...
> Connected to bbc.co.uk.
> Escape character is '^]'.
> 
> 
> Regards,
> 
> 
> Paul Esson  |  Redstor Limited
> t  +44 (0)118 951 5235  |   m  +44 (0)776 690 6514
> e  paul.es...@redstor.com
> www.redstor.com
> 
> 
> 
> 
> 
> -Original Message-
> From: Rodney W. Grimes <freebsd-...@pdx.rh.cn85.dnsmgr.net>
> Sent: 25 April 2018 22:31
> To: Paul Esson <paul.es...@redstor.com>
> Cc: freebsd-virtualization@freebsd.org
> Subject: Re: bhyve networking
> 
> > Hi Rod,
> > Can you share a command line for that? 
> Its not just a single command, but you want these in /etc/rc.conf of the 
> GUEST:
> network_interfaces="lo0"
> cloned_interfaces="vlan48"
> ifconfig_lo0="  inet 127.0.0.1  netmask 0xff00"
> ifconfig_vtnet0="   up"
> ifconfig_vlan48="   inet 192.168.48.38  netmask 0xff00 vlan 48 
> vlandev vtnet0"
> 
> That may be your issue... is your vtnet0 "up" in the guest.
> 
> It would help a whole lot to share more of the info about your system, from 
> commands, not from "vm-bhyve" settings.
> Like
>   ifconfig -a
> on the host and the guest would be a starting point.
> 
> > I also tried presenting an access port from my switch on a specific VLAN - 
> > not trimmed.
> Trimmed?  You mean you set the switch port to untagged mode, and had the 
> switch tag/untag the packets to a specific vlan.
> Be sure you also set the default incoming tag at the switch if you did this, 
> some switches do not follow the vlan setting.
> 
> > Would I still have to tag the interface on the guest in that scenario?
> No.  If I understand what I think you meant by trimmed.
> 
> > Regards,
> > 
> > 
> > Paul Esson | Redstor Limited
> > t +44 (0)118 951 5235
> > m +44 (0)776 690 6514
> > e paul.es...@redstor.com
> > 
> > 
> > From: Rodney W. Grimes <freebsd-...@pdx.rh.cn85.dnsmgr.net>
> > Sent: Wednesday, April 25, 2018 9:33:57 PM
> > To: Paul Esson
> > Cc: Harry Schmalzbauer; freebsd-virtualization@freebsd.org
> > Subject: Re: bhyve networking
> > 
> > [ Charset windows-1252 unsupported, converting... ]
> > > Hi Harry,
> > > I?m simply using the ?vm? utility as in
> > >
> > > vm switch create public
> > > vm switch add public igb1
> > >
> > > That must make underlying calls to if config or equivalent as the bridge 
> > > and tap interfaces are created automatically.
> > >
> > > The vm template file has these relevant parameters
> > >
> > > network0_type=?virtio-net?
> > > network0_switch=?public?
> > >
> > > I?ve done nothing to the igb1 interface other than connect it to a 
> > > physical switch on the appropriate VLAN.
> > 
> > How have you configured your vtnet devices inside the guest?  If your 
> > pass a "trunked" ethernet device to a guest the guest is going to need to 
> > run vlan decapuslation.  I do this here, and it works fine.
> > 
> > vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 
> > mtu 1500
> > options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
> > ether 58:9c:fc:0e:8b:ec
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > media: Ethernet 10Gbase-T 
> > status: active
> > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> > options=63<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> > inet 127.0.0.1 netmask 0xff00
> > inet6 ::1 prefixlen 128
> > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
&

Re: bhyve networking

[Rodney W. Grimes]

   root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> ifmaxaddr 0 port 7 priority 128 path cost 200
> member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> ifmaxaddr 0 port 2 priority 128 path cost 2
> 
> tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 
> 1500
> description: vmnet-testvm-0-public
> options=8
> ether 00:bd:dd:51:0a:00
> hwaddr 00:bd:dd:51:0a:00
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> groups: tap
> Opened by PID 1791
> 
> >From the guest VM I can see that the interface vtnet0 is up and has the 
> >relevant ipv4 address information.  However, I cannot communicate with any 
> >other computer on the guest subnet or beyond.

Guest may be up, but the host interface is in state down.

> vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 
> mtu 1500
> options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
> ether 58:9c:fc:08:4a:20
> hwaddr 58:9c:fc:08:4a:20
> inet xx.xxx.xxx.xx netmask 0xff80 broadcast xx.xxx.xxx.xx
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T 
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=63<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> inet 127.0.0.1 netmask 0xff00
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> groups: lo
> 
> The guest configuration file has the following network details
> network0_type="virtio-net"
> network0_switch="public"
> 
> >From the vm-bhyve.log I see the following
> Apr 26 07:59:23: initialising
> Apr 26 07:59:23:  [loader: bhyveload]
> Apr 26 07:59:23:  [uefi: no]
> Apr 26 07:59:23:  [cpu: 1]
> Apr 26 07:59:23:  [memory: 256M]
> Apr 26 07:59:23:  [hostbridge: standard]
> Apr 26 07:59:23:  [com ports: com1]
> Apr 26 07:59:23:  [uuid: 417cfb63-491f-11e8-949b-246e96b461cc]
> Apr 26 07:59:23:  [utctime: no]
> Apr 26 07:59:23:  [debug mode: no]
> Apr 26 07:59:23:  [primary disk: disk0]
> Apr 26 07:59:23:  [primary disk dev: sparse-zvol]
> Apr 26 07:59:23: generated static mac 58:9c:fc:08:4a:20 (based on 
> 'testvm:0:1524725963:0')
> Apr 26 07:59:23: initialising network device tap0
> Apr 26 07:59:23: adding tap0 -> bridge0 (public)
> Apr 26 07:59:23: booting
> 
> Should I have to supply ipv4 details anywhere other than the guest's own 
> vtnet0 interface?  If I re-configure the switch to remove the igb1 interface 
> and add igb0 instead, then change the guest ipv4 address details to the 
> management network (172.16.92.0/24), I can connect to other computers on that 
> subnet and beyond.  
> 
> vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 
> mtu 1500
> options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
> ether 58:9c:fc:08:4a:20
> hwaddr 58:9c:fc:08:4a:20
> inet 172.16.92.21 netmask 0xff80 broadcast 172.16.92.127
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T 
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=63<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> inet 127.0.0.1 netmask 0xff00
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> groups: lo
> $ ping 172.16.92.11
> PING 172.16.92.11 (172.16.92.11): 56 data bytes
> 64 bytes from 172.16.92.11: icmp_seq=0 ttl=64 time=0.416 ms
> 64 bytes from 172.16.92.11: icmp_seq=1 ttl=64 time=0.371 ms
> 64 bytes from 172.16.92.11: icmp_seq=2 ttl=64 time=0.369 ms
> --- 172.16.92.11 ping statistics ---
> 3 packets transmitted, 3 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.369/0.385/0.416/0.022 ms
> 
> $ telnet bbc.co.uk 443
> Trying 151.101.192.81...
> Connected to bbc.co.uk.
> Escape character is '^]'.
> 
> 
> Regards,
> 
> 
> Paul Esson  |  Redstor Limited
> t  +44 (0)118 951 5235  |   m  +44 (0)776 690 6514
> e  paul.es...@redstor.com
> www.redstor.com
> 
> 
> 
> 
> 
> -Original Message-
> From: Rodney W. Grimes <freebsd-...@pdx.rh.cn85.dnsmgr.net> 
> Sent: 25 Apr

Re: bhyve networking

[Rodney W. Grimes]

> Hi Rod,
> Can you share a command line for that? 
Its not just a single command, but you want these in /etc/rc.conf of the GUEST:
network_interfaces="lo0"
cloned_interfaces="vlan48"
ifconfig_lo0="  inet 127.0.0.1  netmask 0xff00"
ifconfig_vtnet0="   up"
ifconfig_vlan48="   inet 192.168.48.38  netmask 0xff00 vlan 48 
vlandev vtnet0"

That may be your issue... is your vtnet0 "up" in the guest.

It would help a whole lot to share more of the info about
your system, from commands, not from "vm-bhyve" settings.
Like
ifconfig -a
on the host and the guest would be a starting point.

> I also tried presenting an access port from my switch on a specific VLAN - 
> not trimmed.
Trimmed?  You mean you set the switch port to untagged mode,
and had the switch tag/untag the packets to a specific vlan.
Be sure you also set the default incoming tag at the switch if you did this,
some switches do not follow the vlan setting.

> Would I still have to tag the interface on the guest in that scenario?
No.  If I understand what I think you meant by trimmed.

> Regards,
> 
> 
> Paul Esson | Redstor Limited
> t +44 (0)118 951 5235
> m +44 (0)776 690 6514
> e paul.es...@redstor.com
> 
> 
> From: Rodney W. Grimes <freebsd-...@pdx.rh.cn85.dnsmgr.net>
> Sent: Wednesday, April 25, 2018 9:33:57 PM
> To: Paul Esson
> Cc: Harry Schmalzbauer; freebsd-virtualization@freebsd.org
> Subject: Re: bhyve networking
> 
> [ Charset windows-1252 unsupported, converting... ]
> > Hi Harry,
> > I?m simply using the ?vm? utility as in
> >
> > vm switch create public
> > vm switch add public igb1
> >
> > That must make underlying calls to if config or equivalent as the bridge 
> > and tap interfaces are created automatically.
> >
> > The vm template file has these relevant parameters
> >
> > network0_type=?virtio-net?
> > network0_switch=?public?
> >
> > I?ve done nothing to the igb1 interface other than connect it to a physical 
> > switch on the appropriate VLAN.
> 
> How have you configured your vtnet devices inside the guest?  If your pass a 
> "trunked" ethernet device
> to a guest the guest is going to need to run vlan decapuslation.  I do this 
> here, and it works fine.
> 
> vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 
> mtu 1500
> options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
> ether 58:9c:fc:0e:8b:ec
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T 
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=63<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet 127.0.0.1 netmask 0xff00
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> groups: lo
> vlan48: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> ether 58:9c:fc:0e:8b:ec
> inet 192.168.48.38 netmask 0xff00 broadcast 192.168.48.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T 
> status: active
> vlan: 48 vlanpcp: 0 parent interface: vtnet0
> groups: vlan
> 
> ...
> >
> > Bez?glich Paul Esson's Nachricht vom 25.04.2018 20:44 (localtime):
> > > Hi Folks,
> > >
> > > I'm struggling with networking when using vm-bhyve on FreeBSD 
> > > 11.1-RELEASE. I have two NICs and have configured the first (igb0) on a 
> > > management network and want to use the second (igb1) for VMs.  However, I 
> > > can't get any VM to communicate through the virtual switch if I have igb1 
> > > added to it.  If I take the NIC out of the switch and configure an ipv4 
> > > address on it I can reach other hosts on the relevant subnet so I believe 
> > > the poet set-up is valid.  If I replace igb1 in the switch with igb0, I 
> > > can then configure VMs on my management network and they have network 
> > > connectivity.  Can I only use
> >
> > Hello,
> >
> > a example of the command you used was nice.
> > I guess you're using if_bridge(4) ? the example would clarify.
> > But there's ng_bridge(4) and vale(4) also, and others are using
> > OpenVSwitch...
> >
> > >  an interface that has an IP address configured at the host level before 
> > > adding it to the switch?   I've used other
> >
> > No.
> > 

Re: bhyve networking

[Rodney W. Grimes]

[ Charset windows-1252 unsupported, converting... ]
> Hi Harry,
> I?m simply using the ?vm? utility as in
> 
> vm switch create public
> vm switch add public igb1
> 
> That must make underlying calls to if config or equivalent as the bridge and 
> tap interfaces are created automatically.
> 
> The vm template file has these relevant parameters
> 
> network0_type=?virtio-net?
> network0_switch=?public?
> 
> I?ve done nothing to the igb1 interface other than connect it to a physical 
> switch on the appropriate VLAN.

How have you configured your vtnet devices inside the guest?  If your pass a 
"trunked" ethernet device
to a guest the guest is going to need to run vlan decapuslation.  I do this 
here, and it works fine.

vtnet0: flags=8943 metric 0 mtu 
1500
options=80028
ether 58:9c:fc:0e:8b:ec
nd6 options=29
media: Ethernet 10Gbase-T 
status: active
lo0: flags=8049 metric 0 mtu 16384
options=63
inet 127.0.0.1 netmask 0xff00 
inet6 ::1 prefixlen 128 
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
nd6 options=21
groups: lo 
vlan48: flags=8843 metric 0 mtu 1500
ether 58:9c:fc:0e:8b:ec
inet 192.168.48.38 netmask 0xff00 broadcast 192.168.48.255 
nd6 options=29
media: Ethernet 10Gbase-T 
status: active
vlan: 48 vlanpcp: 0 parent interface: vtnet0
groups: vlan 

...
> 
> Bez?glich Paul Esson's Nachricht vom 25.04.2018 20:44 (localtime):
> > Hi Folks,
> >
> > I'm struggling with networking when using vm-bhyve on FreeBSD 11.1-RELEASE. 
> > I have two NICs and have configured the first (igb0) on a management 
> > network and want to use the second (igb1) for VMs.  However, I can't get 
> > any VM to communicate through the virtual switch if I have igb1 added to 
> > it.  If I take the NIC out of the switch and configure an ipv4 address on 
> > it I can reach other hosts on the relevant subnet so I believe the poet 
> > set-up is valid.  If I replace igb1 in the switch with igb0, I can then 
> > configure VMs on my management network and they have network connectivity.  
> > Can I only use
> 
> Hello,
> 
> a example of the command you used was nice.
> I guess you're using if_bridge(4) ? the example would clarify.
> But there's ng_bridge(4) and vale(4) also, and others are using
> OpenVSwitch...
> 
> >  an interface that has an IP address configured at the host level before 
> > adding it to the switch?   I've used other
> 
> No.
> But the interface has to be in promisc mode. And some offloading
> functions must be disabled, but in case of if_bridge(4), it's done
> automagically (and reverted if you remove the interface again).
> 
> Hth,
> 
> -harry
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
> 
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"