Re: core dumps running in bhyve

[Fabian Freyer]

CCing freebsd-virtualization@, because they might know more about this.

Am 25.12.2018 um 02:24 schrieb Chuck Tuffli:

Using the latest bhyve, I'm seeing core dumps in the guest when running:
 nvmecontrol identify nvme0
against the emulated NVMe drive. The location of the core dump changes
from run to run, but I suspect the root cause is a memory corruption
caused by the transfer of the Identify data (4KB) back to the guest.
This transfer of data is actually a memcpy to an address returned from
vm_map_gpa() based on the physical address provided by the guest.

Based on the signature of one of the core dumps, I modified
nvmecontrol to always pass a 4KB aligned buffer to the driver instead
of the (typically) unaligned address of the structure on the stack.
With this change, nvmecontrol in the guest no longer core dumps. What
I don't understand is why this changes the behavior. Do the addresses
passed to vm_map_gpa() need to be page aligned?


AFAIK vm_map_gpa maps a page, so yes, it needs to be 4k-aligned.


Or did moving the
memory location from the stack to the heap merely mitigate what is
corrupted?

Thoughts?

--chuck
___
freebsd-hack...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: on bhyve statistics

[Fabian Freyer]

On 8/28/18 3:37 PM, Rodney W. Grimes wrote:

Currently, bhyve does not expose any of these statistics. All the stats

available through bhyvectl --get-stats seem to be coming from the VMM,
not from the userspace emulation.



That is correct, byhvectl is a diagnostics tool for getting

information from the kernel/vmm module.

bhyvectl provide stats related to processor vmx/svm from vmm.ko and is the
first thing you want to run for performance regression. It will be nice to
include it as part of bhyve perf tool/dashboard that you are intended to
build.


 From conversations with Peter Grehan he expressed that bhyvectl is
purely a diagnostics tool that should not be depended on by any
other tools.

If you want to do similiar things you should program to the libvmmapi
interface, not bhyvectl.


For context, this is *not* what I'm aiming to do. While bhyvectl just 
prints statistics exposed by the VMM through libvmmapi, I'm specifically 
asking about instrumenting the userland part, bhyve.


Also, libvmmapi does have downstream consumers that are not 
bhyve{,ctl,load}, e.g. grub2-bhyve, bhyve-multiboot[1], maybe even 
xhyve[2] - for performance reasons, I'd prefer scraping metrics through 
libvmmapi over shelling out to bhyvectl.


[1] https://github.com/fubarnetes/bhyve-multiboot
[2] https://github.com/mist64/xhyve
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: on bhyve statistics

[Fabian Freyer]

On 27.08.2018 16:34, Marcelo Araujo wrote:
> Hi Fabian,
> 
> 2018-08-27 22:14 GMT+08:00 Fabian Freyer
>  <mailto:fabian.fre...@physik.tu-berlin.de>>:
[...]
> - are the stats in struct bhyvestats {...} stats exposed or used in any
>   non-obvious way?
> 
> 
> They are most used inside bhyverun by the name "stats" and the purpose
> is not really to collect statistics about the guest vm.

What are they used for? I only see them get increased, never actually
checked:

$ rg stats /usr/src/usr.sbin/bhyve
/usr/src/usr.sbin/bhyve/bhyverun.c
117:struct bhyvestats {
126:} stats;
510:stats.vmexit_bogus++;
521:stats.vmexit_reqidle++;
530:stats.vmexit_hlt++;
544:stats.vmexit_pause++;
555:stats.vmexit_mtrap++;
566:stats.vmexit_inst_emul++;


> - architecturally, what would be the best ways to get stats out of the
>   user-space emulations? Off of the top of my head, I could think of the
>   following possibilities:
>   - prometheus exporter
>   - having some socket or pipe to request them
>   - DTrace probes
> 
> 
> I don't know what kind of stats do you need to collect, but based on the
> assumptions you listed above, maybe you can take a look at this project:
> https://github.com/freenas/bhyve-vm-goagent
> 
> bhyve-vm-goagent can easily be extended to collect other information
> from guest. Although I'm not sure if it will be useful for your case,
> looks like you are looking for something less intrusive than
> bhyve-vm-goagent.

Thanks for the pointer - but you're right, I'm looking for
guest-agnostic monitoring on the host.

Fabian
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

on bhyve statistics

[Fabian Freyer]

Hi list,

I'm currently looking at getting the libvirt prometheus exporter[1] to
work with libvirt+bhyve. In its current state this doesn't work because
at least one of the API calls exposed by libvirt isn't implemented by
the libvirt bhyve driver - so I started looking at implementing it.

The first API call in question is virDomainBlockStats[2], which returns
statistics (number of read and written bytes and ops, respectively).

Currently, bhyve does not expose any of these statistics. All the stats
available through bhyvectl --get-stats seem to be coming from the VMM,
not from the userspace emulation. OTOH, I did see that there are *some*
stats being collected in bhyverun.c (see struct bhyvestats {...}
stats;). I can't see how these are exposed though -  a grep of /usr/src
turned up no other uses. Which brings me to the following questions:

- are the stats in struct bhyvestats {...} stats exposed or used in any
  non-obvious way?

- architecturally, what would be the best ways to get stats out of the
  user-space emulations? Off of the top of my head, I could think of the
  following possibilities:
  - prometheus exporter
  - having some socket or pipe to request them
  - DTrace probes

I wouldn't mind implementing any of the above, and so would like to know
which of these (or other options) would be the most acceptable, and
would appreciate some guidance.

CC'ing novel@ for the libvirt side, and grehan@ for the architectural
bhyve questions.

Fabian

[1] https://github.com/kumina/libvirt_exporter
[2] https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockStats
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Checking bhyve supported features (sysctls)

[Fabian Freyer]

On 16 Aug 2018, at 12:16, Matt Churchyard wrote:
I'm looking for better ways to check for bhyve support / available 
features without trying to scan through dmesg output.


There’s a few patches floating around for checking features [1,2] - 
however none of these are yet committed.

For [2], there’s also an RFC mailing list thread [3].

CC’ing novel@, as he wrote those patches.

The hackish way libvirt checks for features right now is to parse error 
messages from invalid invocations of bhyve(8).


[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210111
[2] https://reviews.freebsd.org/D15992
[3] 
https://lists.freebsd.org/pipermail/freebsd-virtualization/2018-June/006536.html

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Segmentation fault in grub-bhyve when trying to boot a Linux guest

[Fabian Freyer]

On 1 May 2018, at 18:41, Mark Raynsford wrote:

> I've recompiled the port with
> WITH_DEBUG=yes (which, unfortunately, took most of the day due to
> having to compile the gcc-6 dependency). Unfortunately, this didn't
> yield a usable backtrace. I'm guessing that the Dwarf debugging
> information isn't compatible with the gdb version that's in base? gdb
> says:

Yes, you need to use gdb from ports for this. Here’s the backtrace:

#0  0x0040d506 in grub_memmove (dest=0x80388f000, src=0x0, n=2047) at 
kern/misc.c:61
#1  0x0049e9eb in grub_memcpy (dest=0x80388f000, src=0x0, n=2048) at 
../include/grub/misc.h:75
#2  0x0049f6b2 in grub_linux_boot () at loader/i386/linux.c:577
#3  0x004271af in grub_loader_boot () at commands/boot.c:162
#4  0x0042720e in grub_cmd_boot (cmd=0x80322d140, argc=0, 
argv=0x80321c138) at commands/boot.c:179
#5  0x004b3a87 in grub_script_execute_cmdline (cmd=0x8032322a8) at 
script/execute.c:927
#6  0x004b353c in grub_script_execute_cmd (cmd=0x8032322a8) at 
script/execute.c:753
#7  0x004b3b85 in grub_script_execute_cmdlist (list=0x803221cc8) at 
script/execute.c:972
#8  0x004b353c in grub_script_execute_cmd (cmd=0x803221cc8) at 
script/execute.c:753
#9  0x004b3ea1 in grub_script_execute (script=0x803232180) at 
script/execute.c:1084
#10 0x004b1084 in grub_normal_parse_line (line=0x803220178 "boot", 
getline=0x4a139a ) at script/main.c:35
#11 0x004a1448 in grub_cmdline_run (nested=0) at normal/main.c:477
#12 0x004a10ad in grub_enter_normal_mode (config=0x803266100 
"(host)/storage/vm/rosemary/grub.cfg") at normal/main.c:320
#13 0x004a1154 in grub_cmd_normal (cmd=0x80322ee00, argc=0, argv=0x0) 
at normal/main.c:356
#14 0x0040cd10 in grub_command_execute (name=0x53120c "normal", argc=0, 
argv=0x0) at ../include/grub/command.h:120
#15 0x0040d326 in grub_load_normal_mode () at kern/main.c:216
#16 0x0040d391 in grub_main () at kern/main.c:250
#17 0x00406165 in main (argc=12, argv=0x7fffeb48) at 
kern/emu/main.c:365

I’m pretty sure the src parameter should not be NULL for grub_memcpy.

Fabian

signature.asc
Description: OpenPGP digital signature

Re: Segmentation fault in grub-bhyve when trying to boot a Linux guest

[Fabian Freyer]

On 1 May 2018, at 0:05, Mark Raynsford via freebsd-virtualization wrote:
> I've recently attempted to install a Debian 9.4.0 x86_64 guest. The
> installer ran to completion without issue, and I then rebooted into the
> installed system, again without issue.
>
> I then shut the system down and tried to bring it up...
>
>   pid 71802 (grub-bhyve), uid 0: exited on signal 11 (core dumped)

Is this reproducible? If yes,
* is it still reproducible on a freshly built grub-bhyve from ports with
  debugging symbols (build the port with WITH_DEBUG=yes)?
* is a core file dumped?
* could you grab a backtrace from the core file?

Fabian

signature.asc
Description: OpenPGP digital signature

Re: rumpkernel and bhyve: triple faults

[Fabian Freyer]

On 10 Mar 2018, at 23:46, Martin Lucina wrote:
> On Friday, 09.03.2018 at 18:45, Fabian Freyer wrote:
>> On 6 Mar 2018, at 7:45, Fabian Freyer wrote:
>>> I’m not sure where to go from here. Is this a bug in bhyve(4), should these
>>> values be initialised somehow, or should I patch rumpkernel(7) to skip this 
>>> check
>>> when running on bhyve(4)?
>
> You probably want to use a serial console rather than VGA on bhyve in any
> case, so you'll want to add the appropriate checks to hypervisor.c and
> cons.c.

I’ve started on a patch, but the check should fail if bios_crtc_base is unset
anyways.

>> A build on Linux (which boots fine) shows [bios_com1_base, bios_crtc_base] 
>> not to
>> be uninitialised:
>> 003e3480 g O .bss0002 bios_com1_base
>> 003e44a0 g O .bss0002 bios_crtc_base
>
> When you write "which boots fine", I presume you're referring to booting on
> bhyve?

Yes. The rumprun kernel built on Linux boots fine (including serial output by 
default
on Linux. The one built on FreeBSD triple faults, due to accessing 0x2.

>> Further down the rabbit hole, this goes on in rumprun.o:
>>
>> On Linux, bios_crtc_base is not a local symbol:
>> 0002   O *COM*  0002 bios_crtc_base
>> 0002   O *COM*  0002 bios_com1_base
>>
>> While on FreeBSD, they are marked as local:
>> 0002 l O *COM*  0002 bios_crtc_base
>> 0002 l O *COM*  0002 bios_com1_base
>
> That seems wrong. Can you try and force the toolchain to use the more
> recent GNU ld from devel/binutils and see if that fixes the problem?

I’m using GNU Binutils from devel/binutils:

pkg list binutils | grep /usr/local/bin/ld
/usr/local/bin/ld
/usr/local/bin/ld.bfd
/usr/local/bin/ld.gold

$(x86_64-rumprun-netbsd-gcc -print-prog-name=ld) -v
GNU ld (GNU Binutils) 2.30

Fabian

signature.asc
Description: OpenPGP digital signature

Re: rumpkernel and bhyve: triple faults

[Fabian Freyer]

On 6 Mar 2018, at 7:45, Fabian Freyer wrote:
> Tracking down bios_crtc_base, I find that it’s loaded in
> rumprun/platform/hw/arch/amd64/locore.S:70:
>
>   /* save BIOS data area values */
>   movw BIOS_COM1_BASE, %bx
>   movw %bx, bios_com1_base
>   movw BIOS_CRTC_BASE, %bx
>   movw %bx, bios_crtc_base
>
> Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking the bhyve
> device node in /dev/vmm with xxd(1), I find the words at these addresses to be
> Uninitialised:
>
> 0400:  ..
> 0483:  ..
>
> I’m not sure where to go from here. Is this a bug in bhyve(4), should these
> values be initialised somehow, or should I patch rumpkernel(7) to skip this 
> check
> when running on bhyve(4)?

I’ve chased this bug down a bit further to what I believe is an issue with the
rumprun toolchain I am building on FreeBSD with the misc/rumprun port [1].

objdump -t helloer-rumprun.elf list a number of symbols in the *COM* section, 
which
holds unallocated C external variables [2]:

objdump -t helloer-rumprun.elf | grep \*COM\*
0001 l O *COM*   0001 pic1mask
0004 l O *COM* 0004 pgalloc_totalkb
0004 l O *COM* 0004 pgalloc_usedkb
1000 l O *COM* 0020 multiboot_cmdline
0002 l O *COM* 0002 bios_crtc_base
0001 l O *COM* 0001 pic2mask
0002 l O *COM* 0002 bios_com1_base

As the pagetable in pagetable.s maps the first page as non-present, accessing 
any
of these will result in a fault. I’m pretty sure that these shouldn’t be 
undefined.

A build on Linux (which boots fine) shows these not to be uninitialised:
003e3480 g O .bss   0002 bios_com1_base
003e44a0 g O .bss   0002 bios_crtc_base

Further down the rabbit hole, this goes on in rumprun.o:

On Linux, bios_crtc_base is not a local symbol:
0002   O *COM*  0002 bios_crtc_base
0002   O *COM*  0002 bios_com1_base

While on FreeBSD, they are marked as local:
0002 l O *COM*  0002 bios_crtc_base
0002 l O *COM*  0002 bios_com1_base

Fabian

[1] 
https://svnweb.freebsd.org/ports/head/misc/rumprun/Makefile?view=markup&pathrev=459195
[2] http://man7.org/linux/man-pages/man5/elf.5.html / SHN_COMMON

signature.asc
Description: OpenPGP digital signature

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de added a comment.


  In D14473#306827 , @grehan wrote:
  
  > So a comment on this: in general, api's are not added to FreeBSD that don't 
have base-system clients that use them, or for a good reason. I think this 
falls into the latter but I'd like to cut it down a bit.
  >
  > - can the get_unrestricted_guest() routine be removed ? There is an error 
return on the set, which seems like it can be used to determine if unrestricted 
mode is not available (e.g. that's how bhyve uses the ioctl).
  
  
  Yes, it can.
  
  > - is there a need for vcpu_reset() ? The BSP should be initialized to 
power-on state.. That could be a bug in bhyve and better to have it fixed there.
  
  Not necessarily, as everything in vcpu_reset() could also be accomplished 
with the other callbacks. I don't think bhyverun should call vcpu_reset(), as 
bhyveload sets up registers before. I guess this should happen before 
`loader_main` is called?

REVISION DETAIL
  https://reviews.freebsd.org/D14473

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, rgrimes, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de added 1 blocking reviewer(s): grehan.

REVISION DETAIL
  https://reviews.freebsd.org/D14473

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, rgrimes, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: rumpkernel and bhyve: triple faults

[Fabian Freyer]

Hi Peter,

On 6 Mar 2018, at 16:15, Peter Grehan wrote:
>  Exception 14 is a page fault (SDM Vol3 ch 6.15). The exception type is 
> "fault" which means it is delivered at the address it was detected at.
>
>  This cascaded very quickly into a triple-fault, so it looks like it could 
> possibly be an issue with the stack. One debug tool you do have is to get a 
> register dump on exit, with 'bhyvectl --get-all --vm='.
>
>  For a page-fault, the virtual address that resulted in the fault will be in 
> the CR2 register.

I don’t see a CR2 register in the output of bhyvectl --get-all, I was looking 
for that too.

>  From the code at the faulting address:
>
>  > 00102a50 :
>  >102a50:   push   rbx
>  >102a51:   call   103540 
>  >102a56:   cmpWORD PTR [rip-0x102a5c],0x0# 2 
> 
>
>  It's using RIP-relative addressing here, but objdump seems to think this may 
> be an offset in the current_lwp structure - is it possible that may have an 
> uninitialized value ?

I’m pretty sure it’s tooling that’s displaying something off, since hopper is 
showing me this as

0x00102a56 cmpword [0x2], 0x0

Which is very similar to what r2 is giving me:

;-- cons_init:
0x00102a50  53 push rbx; /arch/x86:43
0x00102a51  e8ea0a call sym.hypervisor_detect  ; /arch/x86:47
0x00102a56  66833da4d5ef.  cmp word [0x0002], 0; /arch/x86:62

>  (I don't believe this has anything to do with VGA).

Maybe I’m off with my analysis of the actual fault here, but how I understand
the source (assuming compilers work as I would expect, which is not always true)
the values here are initialised from values in the bios data area (which is
zeroed out on bhyve):

#define BIOS_COM1_BASE  0x400
#define BIOS_CRTC_BASE  0x463

...

movw BIOS_COM1_BASE, %bx
movw %bx, bios_com1_base
movw BIOS_CRTC_BASE, %bx
movw %bx, bios_crtc_base

...

/*
 * If the BIOS says no CRTC is present use the serial console if
 * available.
 */
if (bios_crtc_base == 0)
prefer_serial = 1;


Here’s my full output from bhyvectl --get-all:

ID  Length  Name
0   128MB   sysmem
Address Length  Segment Offset  Prot  Flags
0   128MB   sysmem  0   RWX
efer[0] 0x0500
cr0[0]  0x80010031
cr3[0]  0x0010b000
cr4[0]  0x2620
dr7[0]  0x0400
rsp[0]  0x00100ff0
rip[0]  0x00102a56
rax[0]  0x
rbx[0]  0x003eaa2b
rcx[0]  0x68622065
rdx[0]  0x20657679
rsi[0]  0x00100fd0
rdi[0]  0x4000
rbp[0]  0x
r8[0]   0x00100fdc
r9[0]   0x00100fd8
r10[0]  0x00100fd4
r11[0]  0x
r12[0]  0x
r13[0]  0x
r14[0]  0x
r15[0]  0x
rflags[0]   0x00010006
ds desc[0]  0x/0x/0xc093
es desc[0]  0x/0x/0xc093
fs desc[0]  0x/0x/0x0001c001
gs desc[0]  0x/0x/0x0001c001
ss desc[0]  0x/0x/0xc093
cs desc[0]  0x/0x/0xa09b
tr desc[0]  0x/0x/0x008b
ldtr desc[0]0x/0x/0x0082
gdtr[0] 0x00378040/0x002f
idtr[0] 0x/0x
cs[0]   0x0008
ds[0]   0x0018
es[0]   0x0018
fs[0]   0x
gs[0]   0x
ss[0]   0x0018
tr[0]   0x
ldtr[0] 0x
cr0_mask[0] 0x6020
cr0_shadow[0]   0x0021
cr4_mask[0] 0xffe8f800
cr4_shadow[0]   0x
cr3_target_count[0] 0x
cr3_target0[0]  0x
cr3_target1[0]  0x
cr3_target2[0]  0x
cr3_target3[0]  0x
pinbased_ctls[0]0x003f
procbased_ctls[0]   0xf51865f2
procbased_ctls2[0]  0x10a2
gla[0]  0xfec41000
gpa[0]  0x
entry_interruption_info[0]  0x
tpr_threshold[0]0x
instruction_error[0]0x
exit_ctls[0]0x0033efff
entry_ctls[0]   0x93ff
host_pat[0] 0x0001050600070406
host_cr0[0] 0x8005003b
host_cr3[0] 0x38045054
host_cr4[0] 0x001726e0
host_rip[0] 0x81435290
host_rsp[0] 0xfe003218d700
vmcs_pointer[0] 0x
vmcs_exit_i

Re: rumpkernel and bhyve: triple faults

[Fabian Freyer]

(un-CC’ing rumpkernel-users@, since this part of the
discussion is getting very bhyve-specific)

On 6 Mar 2018, at 10:18, Rodney W. Grimes wrote:


On 6 Mar 2018, at 9:28, Rodney W. Grimes wrote:


bios_crtc_base would be part of the isa legacy vga
controller card.  Bhyve does not, at this time, or
in the near future expect to have, support for this
legacy device.


I am wrong on this, the framebuffer device does
infact have support for the legacy i/o addresses
that this should point to.  You should see the
"vgaconf" section of the FrameBuffer section
of the bhyve(8) manpage.

I believe you need to be running bhyve with the
uefi bios options, the with CMS version, and
have vgaconf=on to get your code to work as is.


For diskless multiboot kernels I?m going with a
separate userboot.so-compatible loader. Specifying
a UEFI bootrom implicitly resets the CPU.
(See usr.sbin/bhyve/bhyverun.c:960)


Well in that case my original claim that there
is not a legacy isa vga device avaliable would
be correct for this environment, and you should
just eliminate anything that tries to use it,
or make it understand that this device may not
exist.  I am not sure if bhyve maps any of these
legacy addresses if your not using bhyveloader
or uefi bios code.  0x400 and 0x483 being
unmapped could lead to your tripple fault.


According to lib/libvmmapi/vmmapi.c:409, (and some
code / comments in grub2-bhyve), bhyve maps up to
two memory segments: [0, lowmem) and if more than
3 GiB of memory is given, [4GiB, 4GiB + highmem).

According to the comments in
grub2-bhyve/grub-core/kern/emu/bhyve_hostif.c:149
The area from 640kiB to 1MiB is reserved as the
vga hole and BIOS.

0x400 and 0x463 are in the bios data area [1], which is
mapped, but zeroed out:

  40:00  word COM1 port address
  40:63  word Base port address for active 6845 CRT controller

I’m not sure whether it might not be useful for
bhyve to populate the bios data area with the
information that it does have available (e.g. serial
ports).

On a related note, I saw that there is a relevant
GSoC Project idea [2] to be mentored by tychon@ (CC’d)
that mentions VGA/VESA device emulation independent
of the UEFI frame buffer. I’ll try to look into that
more, and possibly submit a proposal :).

Fabian

[1] http://stanislavs.org/helppc/bios_data_area.html
[2] 
https://wiki.freebsd.org/SummerOfCodeIdeas#VGA_emulation_improvements_for_bhyve

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: rumpkernel and bhyve: triple faults

[Fabian Freyer]

On 6 Mar 2018, at 9:28, Rodney W. Grimes wrote:


bios_crtc_base would be part of the isa legacy vga
controller card.  Bhyve does not, at this time, or
in the near future expect to have, support for this
legacy device.


I am wrong on this, the framebuffer device does
infact have support for the legacy i/o addresses
that this should point to.  You should see the
"vgaconf" section of the FrameBuffer section
of the bhyve(8) manpage.

I believe you need to be running bhyve with the
uefi bios options, the with CMS version, and
have vgaconf=on to get your code to work as is.


For diskless multiboot kernels I’m going with a
separate userboot.so-compatible loader. Specifying
a UEFI bootrom implicitly resets the CPU.
(See usr.sbin/bhyve/bhyverun.c:960)

I think deciding to use the serial output (which is
what most of rumpkernel’s cons_init is doing) based
on the hypervisor is probably the right way to go.
Something similar is already done for XEN:

/*
 * If running under Xen use the serial console.
 */
if (hypervisor == HYPERVISOR_XEN)
prefer_serial = 1;


rumprun/platform/hw/arch/x86/cons.c:59:
   649   0 350887182668 vm testing[0]: handled exception vmexit 
at 0x102a56


Therefore, I?m assuming this is the origin of the fault.

Tracking down bios_crtc_base, I find that it?s loaded in
rumprun/platform/hw/arch/amd64/locore.S:70:

/* save BIOS data area values */
movw BIOS_COM1_BASE, %bx
movw %bx, bios_com1_base
movw BIOS_CRTC_BASE, %bx
movw %bx, bios_crtc_base

Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking 
the bhyve
device node in /dev/vmm with xxd(1), I find the words at these 
addresses to be

Uninitialised:

0400:  ..
0483:  ..

Typo here, should this be 0463?

Yes, sorry about that.

Fabian
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

rumpkernel and bhyve: triple faults

[Fabian Freyer]

Hello lists,

I’m currently playing around with getting rump kernels built with rumpkernel(7) 
running on FreeBSD’s bhyve(4). I’m using a custom boot loader [1] which builds 
on some patches to bhyveload / user boot [2].

To test, I’m using a simple “helloer” unikernel from the tutorial [3]:

#include 
#include 
int
main()
{

printf("Hello, Rumprun ... I'm feeling tired\n");
sleep(2);
printf("much better!\n");
return 0;
}

I’m compiling this for the hw_virtio platform:

x86_64-rumprun-netbsd-gcc -o helloer-rumprun helloer.c
rumprun-bake hw_virtio helloer-rumprun.elf helloer-rumprun

Starting this kernel in bhyve gives me the following VM Exit:


From the Intel SDM 3D, Appendix C [4], VM Exit 2 is a triple fault.

To further debug this, I’m using the ktr(4) support introduced in FreeBSD 
r276098 [5].

I’ve appended a full KTR trace further down in my mail, but I’ll go along the 
trace here with some commentary in execution order, and attempt to annotate the 
source code lines from rumpkernel:

0x104000 is the entry address given in rump kernel’s multi boot header.

00104000 <_start>:

   632   0 350886885990 vm testing[0]: Setting nextrip to 0x104000
   636   0 350886961656 vm testing[0]: Resume execution at 0x104000

This is the cpuid instruction in x86_cpuid which is called from 
hypervisor_detect:

rumprun/platform/hw/arch/x86/x86_subr.c: 91

00102f90 :
  102f90:   49 89 d2movr10,rdx
  102f93:   49 89 c9movr9,rcx
  102f96:   53  push   rbx
  102f97:   89 f8   moveax,edi
  102f99:   0f a2   cpuid

rumprun/platform/hw/arch/x86/hypervisor.c: 38
   637   0 350887061536 vm testing[0]: cpuid 0,0x100fd8
   638   0 350887074874 vm testing[0]: handled cpuid vmexit at 0x102f99
   639   0 350887079146 vm testing[0]: Resume execution at 0x102f9b

rumprun/platform/hw/arch/x86/hypervisor.c: 40
   640   0 350887102692 vm testing[0]: cpuid 0x1,0x100fd8
   641   0 350887117424 vm testing[0]: handled cpuid vmexit at 0x102f99
   642   0 350887121556 vm testing[0]: Resume execution at 0x102f9b

rumprun/platform/hw/arch/x86/hypervisor.c: 51
   643   0 350887143374 vm testing[0]: cpuid 0x4000,0x100fd8
   644   0 350887145218 vm testing[0]: handled cpuid vmexit at 0x102f99
   645   0 350887149146 vm testing[0]: Resume execution at 0x102f9b

The next vexit is in cons_init, directly after the call to hypervisor_detect.

00102a50 :
  102a50:   53  push   rbx
  102a51:   e8 ea 0a 00 00  call   103540 
  102a56:   66 83 3d a4 d5 ef ffcmpWORD PTR [rip-0x102a5c],0x0  
  # 2 

Due to compiler optimisations, the check here isn’t the
(hypervisor == HYPERVISOR_XEN) check directly after the call to 
hypervisor_detect,
but the check (bios_crtc_base == 0) in

rumprun/platform/hw/arch/x86/cons.c:59:
   649   0 350887182668 vm testing[0]: handled exception vmexit at 0x102a56

Therefore, I’m assuming this is the origin of the fault.

Tracking down bios_crtc_base, I find that it’s loaded in
rumprun/platform/hw/arch/amd64/locore.S:70:

/* save BIOS data area values */
movw BIOS_COM1_BASE, %bx
movw %bx, bios_com1_base
movw BIOS_CRTC_BASE, %bx
movw %bx, bios_crtc_base

Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking the bhyve
device node in /dev/vmm with xxd(1), I find the words at these addresses to be
Uninitialised:

0400:  ..
0483:  ..

I’m not sure where to go from here. Is this a bug in bhyve(4), should these
values be initialised somehow, or should I patch rumpkernel(7) to skip this 
check
when running on bhyve(4)?

Fabian

Full KTR trace:
index  cpu timestamptrace
-- ---  -
   673   0 350887364560 vm testing[0]: vcpu state changed from frozen to 
idle
   672   0 350887363766 vm testing[0]: retu 0/14
   671   0 350887362932 vm testing[0]: All vcpus suspended
   670   0 350887357212 vm testing[0]: vcpu state changed from running to 
frozen
   669   0 350887350444 vm testing[0]: returning from vmx_run: exitcode 14
   668   0 350887348806 vm testing: virtual machine successfully suspended 4
   667   0 350887347712 vm testing[0]: triple fault: info1(0x8b08), 
info2(0x8b0e)
   666   0 350887347590 vm testing[0]: Exception 14 delivered: 0x8b0e
   665   0 350887346312 vm testing[0]: handled exception vmexit at 0x102a56
   664   0 350887346178 vm testing[0]: Exception 14 pending
   663   0 350887346060 vm testing[0]: Setting intr_shadow to 0 succeeded
   662   0 350887341876 vm testing[0]: Reflecting exception 14/0 into the 
guest
   661   0 350887339570 vm testing[0]: vm_exit_intinfo: info1(0x8b08)
   660   0 350887315326 vm testi

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de added a comment.


  ping?
  
  Would these changes be welcome in base? If yes, what would still need to be 
done to this review in order to get it into a committable state?

REVISION DETAIL
  https://reviews.freebsd.org/D14473

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, rgrimes, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

bhyve and mmap(2)

[Fabian Freyer]

Hi!

Would be possible to use mmap(2) around libvmmapi to map a file (MAP_PRIVATE) 
and or shared memory segments (MAP_SHARED) into guest physical address space?

Fabian

signature.asc
Description: OpenPGP digital signature

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de updated this revision to Diff 39655.
fabian.freyer_physik.tu-berlin.de added a comment.


  I'm sorry, I seem to have created the previous patch without full context, so 
here it is again, this time with full context.

CHANGES SINCE LAST UPDATE
  https://reviews.freebsd.org/D14473?vs=39653&id=39655

REVISION DETAIL
  https://reviews.freebsd.org/D14473

AFFECTED FILES
  stand/userboot/userboot.h
  usr.sbin/bhyveload/bhyveload.c

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, rgrimes, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
diff --git a/usr.sbin/bhyveload/bhyveload.c b/usr.sbin/bhyveload/bhyveload.c
--- a/usr.sbin/bhyveload/bhyveload.c
+++ b/usr.sbin/bhyveload/bhyveload.c
@@ -560,6 +560,33 @@
 	return (vm_set_desc(ctx, vcpu, reg, base, limit, access));
 }
 
+static int
+cb_vm_get_unrestricted_guest(void* arg, int vcpu, int *retval)
+{
+
+	return (vm_get_capability(ctx, vcpu,
+	VM_CAP_UNRESTRICTED_GUEST, retval));
+}
+
+static int
+cb_vm_set_unrestricted_guest(void* arg, int vcpu, int val)
+{
+	int error, tmp;
+
+	if ((error = cb_vm_get_unrestricted_guest(arg, vcpu, &tmp)))
+		return (error);
+
+	return (vm_set_capability(ctx, vcpu,
+	VM_CAP_UNRESTRICTED_GUEST, val));
+}
+
+static int
+cb_vcpu_reset(void* arg, int vcpu)
+{
+
+	return (vcpu_reset(ctx, vcpu));
+}
+
 static struct loader_callbacks cb = {
 	.getc = cb_getc,
 	.putc = cb_putc,
@@ -593,6 +620,11 @@
 	/* Version 4 additions */
 	.vm_set_register = cb_vm_set_register,
 	.vm_set_desc = cb_vm_set_desc,
+
+	/* Version 5 additions */
+	.vm_set_unrestricted_guest = cb_vm_set_unrestricted_guest,
+	.vm_get_unrestricted_guest = cb_vm_get_unrestricted_guest,
+	.vcpu_reset = cb_vcpu_reset,
 };
 
 static int
@@ -790,7 +822,7 @@
 	addenv("smbios.bios.vendor=BHYVE");
 	addenv("boot_serial=1");
 
-	func(&cb, NULL, USERBOOT_VERSION_4, ndisks);
+	func(&cb, NULL, USERBOOT_VERSION_5, ndisks);
 
 	free(loader);
 	return (0);
diff --git a/stand/userboot/userboot.h b/stand/userboot/userboot.h
--- a/stand/userboot/userboot.h
+++ b/stand/userboot/userboot.h
@@ -41,6 +41,14 @@
  */
 #define	USERBOOT_VERSION_4  4
 
+/*
+ * Version 5 adds callbacks to set unrestricted guest
+ * capabilities and reset the vcpu state. The callback
+ * structure backward compatible (new callbacks have
+ * been added at the tail end).
+ */
+#define	USERBOOT_VERSION_5  5
+
 /*
  * Exit codes from the loader
  */
@@ -210,4 +218,22 @@
 	int	(*vm_set_register)(void *arg, int vcpu, int reg, uint64_t val);
 	int	(*vm_set_desc)(void *arg, int vcpu, int reg, uint64_t base,
 	u_int limit, u_int access);
+
+	/*
+	 * Version 5 additions.
+	 *
+	 * vm_get_unrestricted_guest checks support for the UNRESTRICTED_GUEST
+	 * capability and if supported, sets 'retval'. If unsupported, an error
+	 * code is returned.
+	 *
+	 * vm_set_unrestricted_guest sets the UNRESTRICTED_GUEST capability if
+	 * supported, and returns an error code otherwise.
+	 *
+	 * vcpu_reset initializes the vCPU state as defined in
+	 * Intel Vol 3a, Table 9.1. IA-32 Processor States Following Power-up,
+	 * Reset or INIT
+	 */
+	int	(*vm_get_unrestricted_guest)(void* arg, int vcpu, int *retval);
+	int	(*vm_set_unrestricted_guest)(void* arg, int vcpu, int val);
+	int	(*vcpu_reset)(void* arg, int vcpu);
 };

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de updated this revision to Diff 39653.
fabian.freyer_physik.tu-berlin.de added a comment.
This revision now requires review to proceed.


  pass `USERBOOT_VERSION_5` to `loader_main`.

REPOSITORY
  rS FreeBSD src repository

CHANGES SINCE LAST UPDATE
  https://reviews.freebsd.org/D14473?vs=39622&id=39653

REVISION DETAIL
  https://reviews.freebsd.org/D14473

AFFECTED FILES
  stand/userboot/userboot.h
  usr.sbin/bhyveload/bhyveload.c

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, rgrimes, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
diff --git a/usr.sbin/bhyveload/bhyveload.c b/usr.sbin/bhyveload/bhyveload.c
--- a/usr.sbin/bhyveload/bhyveload.c
+++ b/usr.sbin/bhyveload/bhyveload.c
@@ -560,6 +560,33 @@
 	return (vm_set_desc(ctx, vcpu, reg, base, limit, access));
 }
 
+static int
+cb_vm_get_unrestricted_guest(void* arg, int vcpu, int *retval)
+{
+
+	return (vm_get_capability(ctx, vcpu,
+	VM_CAP_UNRESTRICTED_GUEST, retval));
+}
+
+static int
+cb_vm_set_unrestricted_guest(void* arg, int vcpu, int val)
+{
+	int error, tmp;
+
+	if ((error = cb_vm_get_unrestricted_guest(arg, vcpu, &tmp)))
+		return (error);
+
+	return (vm_set_capability(ctx, vcpu,
+	VM_CAP_UNRESTRICTED_GUEST, val));
+}
+
+static int
+cb_vcpu_reset(void* arg, int vcpu)
+{
+
+	return (vcpu_reset(ctx, vcpu));
+}
+
 static struct loader_callbacks cb = {
 	.getc = cb_getc,
 	.putc = cb_putc,
@@ -593,6 +620,11 @@
 	/* Version 4 additions */
 	.vm_set_register = cb_vm_set_register,
 	.vm_set_desc = cb_vm_set_desc,
+
+	/* Version 5 additions */
+	.vm_set_unrestricted_guest = cb_vm_set_unrestricted_guest,
+	.vm_get_unrestricted_guest = cb_vm_get_unrestricted_guest,
+	.vcpu_reset = cb_vcpu_reset,
 };
 
 static int
@@ -790,7 +822,7 @@
 	addenv("smbios.bios.vendor=BHYVE");
 	addenv("boot_serial=1");
 
-	func(&cb, NULL, USERBOOT_VERSION_4, ndisks);
+	func(&cb, NULL, USERBOOT_VERSION_5, ndisks);
 
 	free(loader);
 	return (0);
diff --git a/stand/userboot/userboot.h b/stand/userboot/userboot.h
--- a/stand/userboot/userboot.h
+++ b/stand/userboot/userboot.h
@@ -41,6 +41,14 @@
  */
 #define	USERBOOT_VERSION_4  4
 
+/*
+ * Version 5 adds callbacks to set unrestricted guest
+ * capabilities and reset the vcpu state. The callback
+ * structure backward compatible (new callbacks have
+ * been added at the tail end).
+ */
+#define	USERBOOT_VERSION_5  5
+
 /*
  * Exit codes from the loader
  */
@@ -210,4 +218,22 @@
 	int	(*vm_set_register)(void *arg, int vcpu, int reg, uint64_t val);
 	int	(*vm_set_desc)(void *arg, int vcpu, int reg, uint64_t base,
 	u_int limit, u_int access);
+
+	/*
+	 * Version 5 additions.
+	 *
+	 * vm_get_unrestricted_guest checks support for the UNRESTRICTED_GUEST
+	 * capability and if supported, sets 'retval'. If unsupported, an error
+	 * code is returned.
+	 *
+	 * vm_set_unrestricted_guest sets the UNRESTRICTED_GUEST capability if
+	 * supported, and returns an error code otherwise.
+	 *
+	 * vcpu_reset initializes the vCPU state as defined in
+	 * Intel Vol 3a, Table 9.1. IA-32 Processor States Following Power-up,
+	 * Reset or INIT
+	 */
+	int	(*vm_get_unrestricted_guest)(void* arg, int vcpu, int *retval);
+	int	(*vm_set_unrestricted_guest)(void* arg, int vcpu, int val);
+	int	(*vcpu_reset)(void* arg, int vcpu);
 };

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de marked 5 inline comments as done.
fabian.freyer_physik.tu-berlin.de added a comment.


  I hope I got them all now

REPOSITORY
  rS FreeBSD src repository

REVISION DETAIL
  https://reviews.freebsd.org/D14473

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de updated this revision to Diff 39622.
fabian.freyer_physik.tu-berlin.de added a comment.


  addressed @grehan's comments regarding style(9)

REPOSITORY
  rS FreeBSD src repository

CHANGES SINCE LAST UPDATE
  https://reviews.freebsd.org/D14473?vs=39619&id=39622

REVISION DETAIL
  https://reviews.freebsd.org/D14473

AFFECTED FILES
  stand/userboot/userboot.h
  usr.sbin/bhyveload/bhyveload.c

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
diff --git a/usr.sbin/bhyveload/bhyveload.c b/usr.sbin/bhyveload/bhyveload.c
--- a/usr.sbin/bhyveload/bhyveload.c
+++ b/usr.sbin/bhyveload/bhyveload.c
@@ -560,6 +560,33 @@
 	return (vm_set_desc(ctx, vcpu, reg, base, limit, access));
 }
 
+static int
+cb_vm_get_unrestricted_guest(void* arg, int vcpu, int *retval)
+{
+
+	return (vm_get_capability(ctx, vcpu,
+	VM_CAP_UNRESTRICTED_GUEST, retval));
+}
+
+static int
+cb_vm_set_unrestricted_guest(void* arg, int vcpu, int val)
+{
+	int error, tmp;
+
+	if ((error = cb_vm_get_unrestricted_guest(arg, vcpu, &tmp)))
+		return (error);
+
+	return (vm_set_capability(ctx, vcpu,
+	VM_CAP_UNRESTRICTED_GUEST, val));
+}
+
+static int
+cb_vcpu_reset(void* arg, int vcpu)
+{
+
+	return (vcpu_reset(ctx, vcpu));
+}
+
 static struct loader_callbacks cb = {
 	.getc = cb_getc,
 	.putc = cb_putc,
@@ -593,6 +620,11 @@
 	/* Version 4 additions */
 	.vm_set_register = cb_vm_set_register,
 	.vm_set_desc = cb_vm_set_desc,
+
+	/* Version 5 additions */
+	.vm_set_unrestricted_guest = cb_vm_set_unrestricted_guest,
+	.vm_get_unrestricted_guest = cb_vm_get_unrestricted_guest,
+	.vcpu_reset = cb_vcpu_reset,
 };
 
 static int
diff --git a/stand/userboot/userboot.h b/stand/userboot/userboot.h
--- a/stand/userboot/userboot.h
+++ b/stand/userboot/userboot.h
@@ -41,6 +41,14 @@
  */
 #define	USERBOOT_VERSION_4  4
 
+/*
+ * Version 5 adds callbacks to set unrestricted guest
+ * capabilities and reset the vcpu state. The callback
+ * structure backward compatible (new callbacks have
+ * been added at the tail end).
+ */
+#define	USERBOOT_VERSION_5  5
+
 /*
  * Exit codes from the loader
  */
@@ -210,4 +218,22 @@
 	int	(*vm_set_register)(void *arg, int vcpu, int reg, uint64_t val);
 	int	(*vm_set_desc)(void *arg, int vcpu, int reg, uint64_t base,
 	u_int limit, u_int access);
+
+	/*
+	 * Version 5 additions.
+	 *
+	 * vm_get_unrestricted_guest checks support for the UNRESTRICTED_GUEST
+	 * capability and if supported, sets 'retval'. If unsupported, an error
+	 * code is returned.
+	 *
+	 * vm_set_unrestricted_guest sets the UNRESTRICTED_GUEST capability if
+	 * supported, and returns an error code otherwise.
+	 *
+	 * vcpu_reset initializes the vCPU state as defined in 
+	 * Intel Vol 3a, Table 9.1. IA-32 Processor States Following Power-up,
+	 * Reset or INIT
+	 */
+	int	(*vm_get_unrestricted_guest)(void* arg, int vcpu, int *retval);
+	int	(*vm_set_unrestricted_guest)(void* arg, int vcpu, int val);
+	int	(*vcpu_reset)(void* arg, int vcpu);
 };

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de marked an inline comment as done.

REPOSITORY
  rS FreeBSD src repository

REVISION DETAIL
  https://reviews.freebsd.org/D14473

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de updated this revision to Diff 39619.
fabian.freyer_physik.tu-berlin.de edited the summary of this revision.
fabian.freyer_physik.tu-berlin.de edited the test plan for this revision.
fabian.freyer_physik.tu-berlin.de added a comment.
This revision now requires review to proceed.


  - updated the comment at the version identifier to correctly list the changes 
introduced by this version
  - added a callback to `vcpu_reset`

REPOSITORY
  rS FreeBSD src repository

CHANGES SINCE LAST UPDATE
  https://reviews.freebsd.org/D14473?vs=39609&id=39619

REVISION DETAIL
  https://reviews.freebsd.org/D14473

AFFECTED FILES
  stand/userboot/userboot.h
  usr.sbin/bhyveload/bhyveload.c

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
diff --git a/usr.sbin/bhyveload/bhyveload.c b/usr.sbin/bhyveload/bhyveload.c
--- a/usr.sbin/bhyveload/bhyveload.c
+++ b/usr.sbin/bhyveload/bhyveload.c
@@ -560,6 +560,30 @@
 	return (vm_set_desc(ctx, vcpu, reg, base, limit, access));
 }
 
+static int
+cb_vm_get_unrestricted_guest(void* arg, int vcpu, int *retval)
+{
+	return (vm_get_capability(ctx, vcpu,
+			VM_CAP_UNRESTRICTED_GUEST, retval));
+}
+
+static int
+cb_vm_set_unrestricted_guest(void* arg, int vcpu, int val)
+{
+	int error, tmp;
+	if ((error = cb_vm_get_unrestricted_guest(arg, vcpu, &tmp)))
+		return error;
+
+	return (vm_set_capability(ctx, vcpu,
+			VM_CAP_UNRESTRICTED_GUEST, val));
+}
+
+static int
+cb_vcpu_reset(void* arg, int vcpu)
+{
+	return vcpu_reset(ctx, vcpu);
+}
+
 static struct loader_callbacks cb = {
 	.getc = cb_getc,
 	.putc = cb_putc,
@@ -593,6 +617,11 @@
 	/* Version 4 additions */
 	.vm_set_register = cb_vm_set_register,
 	.vm_set_desc = cb_vm_set_desc,
+
+	/* Version 5 additions */
+	.vm_set_unrestricted_guest = cb_vm_set_unrestricted_guest,
+	.vm_get_unrestricted_guest = cb_vm_get_unrestricted_guest,
+	.vcpu_reset = cb_vcpu_reset,
 };
 
 static int
diff --git a/stand/userboot/userboot.h b/stand/userboot/userboot.h
--- a/stand/userboot/userboot.h
+++ b/stand/userboot/userboot.h
@@ -41,6 +41,14 @@
  */
 #define	USERBOOT_VERSION_4  4
 
+/*
+ * Version 5 adds callbacks to set unrestricted guest
+ * capabilities and reset the vcpu state. The callback
+ * structure backward compatible (new callbacks have
+ * been added at the tail end).
+ */
+#define	USERBOOT_VERSION_5  5
+
 /*
  * Exit codes from the loader
  */
@@ -210,4 +218,22 @@
 	int	(*vm_set_register)(void *arg, int vcpu, int reg, uint64_t val);
 	int	(*vm_set_desc)(void *arg, int vcpu, int reg, uint64_t base,
 	u_int limit, u_int access);
+
+	/*
+	 * Version 5 additions.
+	 *
+	 * vm_get_unrestricted_guest checks support for the UNRESTRICTED_GUEST
+	 * capability and if supported, sets 'retval'. If unsupported, an error
+	 * code is returned.
+	 *
+	 * vm_set_unrestricted_guest sets the UNRESTRICTED_GUEST capability if
+	 * supported, and returns an error code otherwise.
+	 *
+	 * vcpu_reset initializes the vCPU state as defined in 
+	 * Intel Vol 3a, Table 9.1. IA-32 Processor States Following Power-up,
+	 * Reset or INIT
+	 */
+	int	(*vm_get_unrestricted_guest)(void* arg, int vcpu, int *retval);
+	int	(*vm_set_unrestricted_guest)(void* arg, int vcpu, int val);
+	int	(*vcpu_reset)(void* arg, int vcpu);
 };

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de added a comment.


  I may also add a vcpu_reset callback still, would be nice to have that in as 
well.

INLINE COMMENTS

> userboot.h:45
> +/*
> + * Version 5 adds callbacks to set up long mode and set
> + * unrestricted guest capabilities. The callback structure

Tbh this is no longer true, so I will update the patch to fix the comment

REPOSITORY
  rS FreeBSD src repository

REVISION DETAIL
  https://reviews.freebsd.org/D14473

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp
Cc: imp, freebsd-virtualization-list, #contributor_reviews_base
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[fabian.freyer_physik.tu-berlin.de (Fabian Freyer)]

fabian.freyer_physik.tu-berlin.de created this revision.
fabian.freyer_physik.tu-berlin.de added a project: bhyve.
Herald added subscribers: Contributor Reviews (base), imp.

REVISION SUMMARY
  The current userboot interface sets this capability using 
`vm_setup_freebsd_registers_i386` in lib/libvmmapi/libvmmapi_freebsd.c. This 
patch adds support for loaders to explicitly query and/or set this capability 
without having to go through the `exec` callback, but set up registers 
themselves.
  
  This change is backwards-compatible as all new callbacks are added to the end 
of the callback structure.

TEST PLAN
  test with a library implementing `loader_main` and calling these callbacks. 
Check whether capability is set.

REPOSITORY
  rS FreeBSD src repository

REVISION DETAIL
  https://reviews.freebsd.org/D14473

AFFECTED FILES
  stand/userboot/userboot.h
  usr.sbin/bhyveload/bhyveload.c

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de
Cc: imp, freebsd-virtualization-list, #contributor_reviews_base
diff --git a/usr.sbin/bhyveload/bhyveload.c b/usr.sbin/bhyveload/bhyveload.c
--- a/usr.sbin/bhyveload/bhyveload.c
+++ b/usr.sbin/bhyveload/bhyveload.c
@@ -560,6 +560,24 @@
 	return (vm_set_desc(ctx, vcpu, reg, base, limit, access));
 }
 
+static int
+cb_vm_get_unrestricted_guest(void* arg, int vcpu, int *retval)
+{
+	return (vm_get_capability(ctx, vcpu,
+			VM_CAP_UNRESTRICTED_GUEST, retval));
+}
+
+static int
+cb_vm_set_unrestricted_guest(void* arg, int vcpu, int val)
+{
+	int error, tmp;
+	if ((error = cb_vm_get_unrestricted_guest(arg, vcpu, &tmp)))
+		return error;
+
+	return (vm_set_capability(ctx, vcpu,
+			VM_CAP_UNRESTRICTED_GUEST, val));
+}
+
 static struct loader_callbacks cb = {
 	.getc = cb_getc,
 	.putc = cb_putc,
@@ -593,6 +611,10 @@
 	/* Version 4 additions */
 	.vm_set_register = cb_vm_set_register,
 	.vm_set_desc = cb_vm_set_desc,
+
+	/* Version 5 additions */
+	.vm_set_unrestricted_guest = cb_vm_set_unrestricted_guest,
+	.vm_get_unrestricted_guest = cb_vm_get_unrestricted_guest,
 };
 
 static int
diff --git a/stand/userboot/userboot.h b/stand/userboot/userboot.h
--- a/stand/userboot/userboot.h
+++ b/stand/userboot/userboot.h
@@ -41,6 +41,14 @@
  */
 #define	USERBOOT_VERSION_4  4
 
+/*
+ * Version 5 adds callbacks to set up long mode and set
+ * unrestricted guest capabilities. The callback structure
+ * backward compatible (new callbacks have been added at
+ * the tail end).
+ */
+#define	USERBOOT_VERSION_5  5
+
 /*
  * Exit codes from the loader
  */
@@ -210,4 +218,17 @@
 	int	(*vm_set_register)(void *arg, int vcpu, int reg, uint64_t val);
 	int	(*vm_set_desc)(void *arg, int vcpu, int reg, uint64_t base,
 	u_int limit, u_int access);
+
+	/*
+	 * Version 5 additions.
+	 *
+	 * vm_get_unrestricted_guest checks support for the UNRESTRICTED_GUEST
+	 * capability and if supported, sets 'retval'. If unsupported, an error
+	 * code is returned.
+	 *
+	 * vm_set_unrestricted_guest sets the UNRESTRICTED_GUEST capability if
+	 * supported, and returns an error code otherwise.
+	 */
+	int	(*vm_get_unrestricted_guest)(void* arg, int vcpu, int *retval);
+	int	(*vm_set_unrestricted_guest)(void* arg, int vcpu, int val);
 };

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: VMX exit reason=33 and general userboot.so questions

[Fabian Freyer]

Hi Peter,

thanks for your response!

On 21 Feb 2018, at 17:59, Peter Grehan wrote:
>  >exit_reason 33
>
>  From the Intel SDM, vol 3B Appendix C, this error is "VM-entry failure due 
> to invalid guest state".

Yes, I’m currently going through SDM, vol 3B, section 26.3, and

>  These errors can be difficult to debug given the large amount of guest state 
> involved :(

definitely accurately describes the experience. That said, I’m going through 
each check and trying to see what is going wrong.

>  However, looking at the state from your dump:
>
>  > tr desc[0]  0x/0x/0x
>
>  I believe you will have to set this. Here's the comment and relevant code 
> fragment from grub2-bhyve 
> grub-core/kern/emu/bhyve_hostif.c:grub_emu_bhyve_boot32()
>
>   /*
>* XXX TR is pointing to null selector even though we set the
>* TSS segment to be usable with a base address and limit of 0.
>* Has to be 8b or vmenter will fail
>*/
>   desc_access = 0x008b;
>   assert(vm_set_desc(bhyve_ctx, 0, VM_REG_GUEST_TR, 0x1000, 0x67,
>  desc_access) == 0);
>
>  grub2-bhyve has been able to load/boot multiboot images, so I suspect the 
> register settings in grub_emu_bhyve_boot32() are a good place to start from.

Thanks. I’ve tried setting all registers (especially) the segment registers as 
grub2-bhyve does, but I still don’t seem to be able to boot correctly. Fixing 
the TR descriptor didn’t do it.

Here’s a list of the descriptors I set up:

ds desc[0]  0x/0x/0x0093
es desc[0]  0x/0x/0x0093
fs desc[0]  0x/0x/0x0093
gs desc[0]  0x/0x/0x0093
ss desc[0]  0x/0x/0x0093
cs desc[0]  0x/0x/0x009b
tr desc[0]  0x1000/0x0067/0x008b
ldtr desc[0]0x/0x/0x00010082


gdtr points to a default gdt as set up in grub2-bhyve:

static uint16_t bhyve_gdt[] = {
  0x, 0x, 0x, 0x,   /* Null */
  0x, 0x, 0x, 0x,   /* Null #2 */
  0x, 0x, 0x9a00, 0x00cf,   /* code */
  0x, 0x, 0x9200, 0x00cf,   /* data */
  0x, 0x, 0x8900, 0x0080/* tss */
};

gdtr[0] 0x00100161/0x0027
idtr[0] 0x/0x

The rest is set up as by grub2-bhyve:
cs[0]   0x0008
ds[0]   0x0010
es[0]   0x0010
fs[0]   0x0010
gs[0]   0x0010
ss[0]   0x0010
tr[0]   0x
ldtr[0] 0x

I’m assuming entry_ctls[0] = 0x91fb is the VM-entry control, but is it the 
value described by Table 24-7 of the SDM Volume 3B? Is this constant for the 
first entry after running bhyveload?  That is, when writing some compliance 
checks against section 26.3.1, can I just hardcode that value?

I’m starting to dig through the bhyverun code, but I’m still pretty new to VMX 
stuff, so I am ending up with more questions than answers.

Fabian


signature.asc
Description: OpenPGP digital signature

VMX exit reason=33 and general userboot.so questions

[Fabian Freyer]

Hi!

I’m currently writing a userboot.so-compatible boot loader [1] and am slowly 
getting to a point were I want to do some testing with running test kernels in 
bhyve.

At the moment, I’m getting the following error after loading my kernel:

---8< snip
vm exit[0]
reason  VMX
rip 0x001c
inst_length 0
status  0
exit_reason 33
qualification   0x
inst_type   0
inst_error  0
[1][PID] abort  bhyve -H -P -s 0,hostbridge -s 31,lpc -c 1 -m 128M 
testing
--->8 snap

This is the register state I’m setting up before running bhyve:

---8< snip
efer[0] 0x
cr0[0]  0x0021
cr3[0]  0x
cr4[0]  0x
dr7[0]  0x
rsp[0]  0x
rip[0]  0x001c
rax[0]  0x2badb002
rbx[0]  0x00100952
rcx[0]  0x
rdx[0]  0x
rsi[0]  0x
rdi[0]  0x
rbp[0]  0x
r8[0]   0x
r9[0]   0x
r10[0]  0x
r11[0]  0x
r12[0]  0x
r13[0]  0x
r14[0]  0x
r15[0]  0x
rflags[0]   0x0002
ds desc[0]  0x/0x/0xc093
es desc[0]  0x/0x/0xc093
fs desc[0]  0x/0x/0xc093
gs desc[0]  0x/0x/0xc093
ss desc[0]  0x/0x/0xc093
cs desc[0]  0x/0x/0xc09b
tr desc[0]  0x/0x/0x
ldtr desc[0]0x/0x/0x
gdtr[0] 0x/0x
idtr[0] 0x/0x
cs[0]   0x
ds[0]   0x
es[0]   0x
fs[0]   0x
gs[0]   0x
ss[0]   0x
tr[0]   0x
ldtr[0] 0x
[... omitted some, not sure if relevant]
--->8 snap

And here’s a diff of the register state before and after running bhyve:

---8< snip
--- before2018-02-20 22:36:16.001919000 +
+++ after   2018-02-20 22:36:27.442941000 +
--- before2018-02-20 22:36:16.001919000 +
+++ after   2018-02-20 22:36:27.442941000 +
@@ -56 +56 @@
-procbased_ctls[0]  0xb5186572
+procbased_ctls[0]  0xf51865f2
@@ -67 +67 @@
-host_cr3[0]0x
+host_cr3[0]0x000389cac09a
@@ -101,4 +101,4 @@
-exit_reason[0] 0
-rtc nvram[000]: 0x05
-rtc time 0x5: Thu Jan 01 00:00:05 1970
-Capability "hlt_exit" is not set on vcpu 0
+exit_reason[0] 0x8021
+rtc nvram[000]: 0x26
+rtc time 0x5a8ca2ea: Tue Feb 20 22:36:26 2018
+Capability "hlt_exit" is set on vcpu 0
@@ -106 +106 @@
-Capability "pause_exit" is not set on vcpu 0
+Capability "pause_exit" is set on vcpu 0
@@ -109 +109 @@
-active cpus:(none)
+active cpus:0
@@ -125 +125 @@
-number of vm exits for unknown reason  0
+number of vm exits for unknown reason  1
@@ -128 +128 @@
-number of vm exits handled in userspace0
+number of vm exits handled in userspace1
@@ -131 +131 @@
-vcpu total runtime 0
+vcpu total runtime 11904
@@ -165,3 +165,3 @@
-Number of vpid invalidations done  0
-vcpu migration across host cpus0
-total number of vm exits   0
+Number of vpid invalidations done  1
+vcpu migration across host cpus1
+total number of vm exits   1
--->8 snap

The code at that RIP point also looks correct:

xxd -s 0x001c -l 5 /dev/vmm/testing
001c: bc00 2000 00

Which decompiles to:
001c: bc0020 mov esp,0x2000

I’ve been looking at what userboot.so, grub2-bhyve and 
vm_setup_freebsd_registers is doing. However, I left all registers that don’t 
have a defined state in the multiboot specification [2] in the state they are.

How would I best start debugging this?

Fabian

[1] https://github.com/fabianfreyer/bhyve-multiboot/tree/multiboot/info
[2] https://www.gnu.org/software/grub/manual/multiboot/multiboot.html


signature.asc
Description: OpenPGP digital signature

Re: grub-bhyve: support overriding just --root flag

[Fabian Freyer]

On 12 Nov 2017, at 0:46, Allan Jude wrote:
> Does libvirt support using the bhyve UEFI-CSM firmware instead? That
> would let the VM boot using the native grub installed inside the VM, and
> avoid this issue entirely. It also makes starting a bhyve a single
> command instead of 2.

Yes it does[1]. Also be aware that bootloader_args has some quoting issues.

CC’ing novel@ as he does a lot of the libvirt+bhyve driver stuff.

Fabian.

[1] https://libvirt.org/drvbhyve.html#uefi


signature.asc
Description: OpenPGP digital signature

Re: new ports for UEFI firmware

[Fabian Freyer]

Hi,

there are now two ports:

  sysutils/uefi-edk2-bhyve
  sysutils/uefi-edk2-bhyve-csm

which build with and without CSM built in, to enable both CSM and non-CSM
firmwares to be installed in parallel.

The firmwares are both installed to /usr/local/share/uefi-firmware/.

Thanks to Roman Bogorodskiy for creating the slave port and Michael Dexter for
providing feedback on the original port.

Fabian



signature.asc
Description: OpenPGP digital signature

new port for UEFI firmware

[Fabian Freyer]

Hi,

there is now a port for the UEFI firmware for bhyve:
http://www.freshports.org/sysutils/uefi-edk2-bhyve

Since it depends on gcc48 to build, the builds take quite some time, but
hopefully there will be a binary package soon.

I've updated https://wiki.freebsd.org/bhyve/UEFI to reflect this, but
there is probably quite some documentation out there that could still be
updated.

Fabian

-- 
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?



signature.asc
Description: OpenPGP digital signature

Re: bhyve graphics support

[Fabian Freyer]

Hi everybody,

I've added some code to support VNC Authentication (as much as a 16-byte
DES challenge response could be seen as Authentication) as per
https://tools.ietf.org/html/rfc6143#page-10.

It's my first time working with this reviews thing - I'm a bit unsure
how to set the diff to be based off the projects/ instead of head/, so
if I still need to change something, I would.

https://reviews.freebsd.org/D7029

It's been tested to work with TightVNC Viewer version 1.3.9. Not sure
about other ones - the RFC is a bit unclear here and there.

Fabian Freyer
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Fwd: [GSOC 2016] Introduction and proposal

[Fabian Freyer]

Hi,

I've written this post to the freebsd-hackers@ mailing list, but I think it 
belongs here better. While I appreciate it's very short before the GSOC 
application deadline, I'd like to ask how the resonance to such a proposal 
would be, and how the chances to finding a mentor for that project stand.

Fabian Freyer
--- Begin Message ---
Hi!

First of all, I would like to introduce myself:
My name is Fabian Freyer, I am a physics student at the TU Berlin where
I have been working as a FreeBSD Sysadmin for about 3 years.
I am fluent in C and python and am interested in OS development; in fact
I have been working on a toy operating system and boot loader as a side
project for a while now. I enjoy playing CTFs and am working on my
skills at reading and writing x86, amd64 and arm assembly.
I have not yet participated in a GSOC due to other projects, but would
like to participate this year.

I started using bhyve with libvirt around a year ago to play around
with, and have been using it as my main supervisor on server and
development systems since; especially the debugging interface has proven
useful to me on many occasions. I could imagine working on one or more
of the following projects, since these are all features I would like to
see in bhyve:
1) Implementing suspend / resume / snapshot behaviour for bhyve vm's
2) Going further from 1) implementing porcelain for migrating
zvol-backed bhyve guests using zfs-send and moving state between hypervisors
3) Working on better libvirt support for bhyve.

However, I haven't yet had much contact with hypervisor code. I have
started looking into SimpleVisor [1] and the usr.sbin/bhyve/ source.
I would greatly appreciate some guidance in that direction.

Fabian Freyer

Links:
[1] - https://github.com/ionescu007/SimpleVisor

-- 
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
___
freebsd-hack...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
--- End Message ---


signature.asc
Description: OpenPGP digital signature