Re: bhyve: corrupting zfs pools?
On 04/06/2015 07:37, Neel Natu wrote: > Ok, there are some differences in our systems. The interesting ones > are number of ASIDs (64 versus 65536), flush-by-asid capability, > vmcb-clean capability and the number of cores. > > I was able to mimic all of these on my Opteron but still wasn't able > to reproduce the issue. I am going to get a Sempron tomorrow which > belongs to the same processor family as the Athlon II so hoping that > it is easier to repro. > > BTW does this happen consistently on your system? Yes, it happened every time I tried that scenario. Thank you very much! If there is anything else that I can do on my side to help the investigation just let me know. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: corrupting zfs pools?
Hi Andriy, On Wed, Jun 3, 2015 at 3:01 AM, Andriy Gapon wrote: > On 03/06/2015 00:40, Neel Natu wrote: >> Perhaps, but I wasn't able to repro this. I tried your recipe to repro >> on two systems running -current at r283917. >> >> - Intel Sandybridge server: Xeon E52650 with 8 cores/16 threads and 64GB >> memory >> - hdd1/hdd2 on UFS >> - hdd1/hdd2 on ZFS >> >> - AMD Opteron server: Opteron 6230 with 8 cores and 16GB memory >> - hdd1/hdd2 on UFS >> >> Can you provide some more details about your setup? I can then try to >> repro on a system similar to your setup. >> >> - uname -a > > FreeBSD trant 11.0-CURRENT FreeBSD 11.0-CURRENT #134 r283188+ab4f83f(devel): > Fri > May 22 15:55:27 EEST 2015 avg@trant:/usr/obj/usr/src/sys/TRANT amd64 > > This is not a pure FreeBSD, there are some local changes, but none to vmm or > VM. > >> - sysctl hw.model > > hw.model: AMD Athlon(tm) II X2 250 Processor > >> - sysctl hw.ncpu > > hw.ncpu: 2 > >> - sysctl hw.physmem > > hw.physmem: 8029335552 > >> - sysctl hw.vmm > > hw.vmm.npt.pmap_flags: 507 > hw.vmm.svm.num_asids: 64 > hw.vmm.svm.disable_npf_assist: 0 > hw.vmm.svm.features: 15 > hw.vmm.svm.vmcb_clean: 959 > hw.vmm.vmx.vpid_alloc_failed: 0 > hw.vmm.vmx.posted_interrupt_vector: -1 > hw.vmm.vmx.cap.posted_interrupts: 0 > hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 > hw.vmm.vmx.cap.invpcid: 0 > hw.vmm.vmx.cap.monitor_trap: 0 > hw.vmm.vmx.cap.unrestricted_guest: 0 > hw.vmm.vmx.cap.pause_exit: 0 > hw.vmm.vmx.cap.halt_exit: 0 > hw.vmm.vmx.initialized: 0 > hw.vmm.vmx.cr4_zeros_mask: 0 > hw.vmm.vmx.cr4_ones_mask: 0 > hw.vmm.vmx.cr0_zeros_mask: 0 > hw.vmm.vmx.cr0_ones_mask: 0 > hw.vmm.ept.pmap_flags: 0 > hw.vmm.vrtc.flag_broken_time: 1 > hw.vmm.ppt.devices: 0 > hw.vmm.iommu.initialized: 0 > hw.vmm.bhyve_xcpuids: 4 > hw.vmm.topology.cpuid_leaf_b: 1 > hw.vmm.topology.cores_per_package: 1 > hw.vmm.topology.threads_per_core: 1 > hw.vmm.create: beavis > hw.vmm.destroy: beavis > hw.vmm.force_iommu: 0 > hw.vmm.trace_guest_exceptions: 0 > hw.vmm.ipinum: 251 > hw.vmm.halt_detection: 1 > >> - host filesystem underlying hdd1 and hdd2 > > It's ZFS. > Ok, there are some differences in our systems. The interesting ones are number of ASIDs (64 versus 65536), flush-by-asid capability, vmcb-clean capability and the number of cores. I was able to mimic all of these on my Opteron but still wasn't able to reproduce the issue. I am going to get a Sempron tomorrow which belongs to the same processor family as the Athlon II so hoping that it is easier to repro. BTW does this happen consistently on your system? best Neel > Thank you! > > -- > Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Where are the VCPU threads created?
On Wed, Jun 3, 2015 at 8:17 PM, Peter Grehan wrote: >> I see that in the main() function of bhyverun.c it adds CPU0 by >> calling fbsdrun_addcpu(ctx, 0, 0, rip). But I can't find where the >> other VCPUs are added. Or if the 'guest_ncpus' variable is ever used >> (except for error checks) > > > Have a look at bhyve/spinup_ap.c:spinup_ap() which is called when a the vmm > kernel module detects a SIPI message being sent to the local APIC and forces > a vm-exit to userspace. Hi Peter, Thank you, that answered my question. Another thing: Are these threads (the ones that run the VCPUs) ever destroyed and recreated? In the timespan between the moment they are created (when the guest starts) and when the guest is shut down, is there any chance of the threads ending/being killed and others created to take their place? Or do you end up with the same threads? I am trying to add some information to the VCPU thread at the moment it is created by bhyve, and want that information to be available for the entirety of the guest's lifespan. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On Wed, Jun 3, 2015 at 9:18 AM, Andriy Gapon wrote: > On 03/06/2015 18:53, John-Mark Gurney wrote: > > JIMO, bhyveload(8) could be just merged into bhyve(8) and the latter should > behave like vmrun.sh does. bhyve(8) should retain an option to run a > preloaded > kernel, so that things like bhyve-grub keep working. > I understand why bhyveload(8) and bhyve(8) exists. However, from a usability perspective, the fact that I need to invoke two commands to start a VM is unfortunate. >From a end-user usability standpoint, this is no good. vmrun.sh is not bad for simple usage, but its limitations become apparent quite quickly. For anyone doing serious stuff with bhyve, you end up having to write your own replacement for vmrun.sh. I've done it, and I am sure others have as well. What I would like to see is a single command that I can type with all the command-line options to start a bhyve VM, instead of multiple commands that need to be stitched together via a script. For example, if bhyve(8) could be enhanced with a flag "--loader" (or choose another flag name that you prefer): (1) If "--loader" is not specified, then by default, bhyve uses bhyveload (2) If "--loader" is specified, then valid options are bhyveload and grub-bhyve (3) bhyve(8) will internally fork the loader specified in (1) or (2) (4) If (3) fails, then stop. If (3) succeeds, then bhyve(8) will continue to start the VM I would also like to see: -> command-line option parsing for bhyveload and grub-bhyve changed, so that they silently ignore command-line options that they don't grok. That would make it easier to implement (3), since bhyve(8) could pass its full command-line arguments to the forked loader. I don't have time to work on this, but it would be nice if this could be implemented. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Where are the VCPU threads created?
Hi Stefan, I see that in the main() function of bhyverun.c it adds CPU0 by calling fbsdrun_addcpu(ctx, 0, 0, rip). But I can't find where the other VCPUs are added. Or if the 'guest_ncpus' variable is ever used (except for error checks) Have a look at bhyve/spinup_ap.c:spinup_ap() which is called when a the vmm kernel module detects a SIPI message being sent to the local APIC and forces a vm-exit to userspace. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Where are the VCPU threads created?
I see that in the main() function of bhyverun.c it adds CPU0 by calling fbsdrun_addcpu(ctx, 0, 0, rip). But I can't find where the other VCPUs are added. Or if the 'guest_ncpus' variable is ever used (except for error checks) ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On 03/06/2015 18:53, John-Mark Gurney wrote:
> Andriy Gapon wrote this message on Wed, Jun 03, 2015 at 10:10 +0300:
>> On 03/06/2015 02:04, John-Mark Gurney wrote:
>>> Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300:
I guess you are running bhyve through the shell script vmrun.sh?
I am doing everything by hand.
>>>
>>> Correct:
>>> sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0
>>>
>>> It's nice.. shutdown -r now and shutdown -p now both work exactly as
>>> you'd expect them to... :)
>>
>> Yes, it's quite convenient. The only drawback is that currently it can't
>> pass
>> multiple -d options to bhyveload and I need that from time to time.
>
> Hmm... Looking at the script, it looks like you can (at least on HEAD):
> d)
> eval "disk_dev${disk_total}=\"${OPTARG}\""
> disk_total=$(($disk_total + 1))
This was only half of the equation, please see this:
https://reviews.freebsd.org/D2723
>> Probably not hard to change that.
>
> It'd also be nice to allow you to switch the default from virtio-blk
> to ahci when you're image doesn't support it..
>
> Hmm.. It wouldn't be hard to try to use environment vars for the defaults
> either...
>
> Maybe it's time to move vmrun.sh to bhyverun, install it and give
> it a proper man page?
JIMO, bhyveload(8) could be just merged into bhyve(8) and the latter should
behave like vmrun.sh does. bhyve(8) should retain an option to run a preloaded
kernel, so that things like bhyve-grub keep working.
--
Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
Andriy Gapon wrote this message on Wed, Jun 03, 2015 at 10:10 +0300:
> On 03/06/2015 02:04, John-Mark Gurney wrote:
> > Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300:
> >> I guess you are running bhyve through the shell script vmrun.sh?
> >> I am doing everything by hand.
> >
> > Correct:
> > sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0
> >
> > It's nice.. shutdown -r now and shutdown -p now both work exactly as
> > you'd expect them to... :)
>
> Yes, it's quite convenient. The only drawback is that currently it can't pass
> multiple -d options to bhyveload and I need that from time to time.
Hmm... Looking at the script, it looks like you can (at least on HEAD):
d)
eval "disk_dev${disk_total}=\"${OPTARG}\""
disk_total=$(($disk_total + 1))
> Probably not hard to change that.
It'd also be nice to allow you to switch the default from virtio-blk
to ahci when you're image doesn't support it..
Hmm.. It wouldn't be hard to try to use environment vars for the defaults
either...
Maybe it's time to move vmrun.sh to bhyverun, install it and give
it a proper man page?
--
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve and illumos
Hi Andriy, Has anyone tried to boot an illumos-based OS in bhyve? Any success stories? The Illumos kernel makes some BIOS calls so it can't be booted with grub-bhyve/multiboot - a triple-fault will result on the first BIOS call. It will need the UEFI/CSM firwmare work that's currently being worked on. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
bhyve and illumos
Has anyone tried to boot an illumos-based OS in bhyve? Any success stories? Thanks! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: corrupting zfs pools?
On 03/06/2015 00:40, Neel Natu wrote: > Perhaps, but I wasn't able to repro this. I tried your recipe to repro > on two systems running -current at r283917. > > - Intel Sandybridge server: Xeon E52650 with 8 cores/16 threads and 64GB > memory > - hdd1/hdd2 on UFS > - hdd1/hdd2 on ZFS > > - AMD Opteron server: Opteron 6230 with 8 cores and 16GB memory > - hdd1/hdd2 on UFS > > Can you provide some more details about your setup? I can then try to > repro on a system similar to your setup. > > - uname -a FreeBSD trant 11.0-CURRENT FreeBSD 11.0-CURRENT #134 r283188+ab4f83f(devel): Fri May 22 15:55:27 EEST 2015 avg@trant:/usr/obj/usr/src/sys/TRANT amd64 This is not a pure FreeBSD, there are some local changes, but none to vmm or VM. > - sysctl hw.model hw.model: AMD Athlon(tm) II X2 250 Processor > - sysctl hw.ncpu hw.ncpu: 2 > - sysctl hw.physmem hw.physmem: 8029335552 > - sysctl hw.vmm hw.vmm.npt.pmap_flags: 507 hw.vmm.svm.num_asids: 64 hw.vmm.svm.disable_npf_assist: 0 hw.vmm.svm.features: 15 hw.vmm.svm.vmcb_clean: 959 hw.vmm.vmx.vpid_alloc_failed: 0 hw.vmm.vmx.posted_interrupt_vector: -1 hw.vmm.vmx.cap.posted_interrupts: 0 hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 hw.vmm.vmx.cap.invpcid: 0 hw.vmm.vmx.cap.monitor_trap: 0 hw.vmm.vmx.cap.unrestricted_guest: 0 hw.vmm.vmx.cap.pause_exit: 0 hw.vmm.vmx.cap.halt_exit: 0 hw.vmm.vmx.initialized: 0 hw.vmm.vmx.cr4_zeros_mask: 0 hw.vmm.vmx.cr4_ones_mask: 0 hw.vmm.vmx.cr0_zeros_mask: 0 hw.vmm.vmx.cr0_ones_mask: 0 hw.vmm.ept.pmap_flags: 0 hw.vmm.vrtc.flag_broken_time: 1 hw.vmm.ppt.devices: 0 hw.vmm.iommu.initialized: 0 hw.vmm.bhyve_xcpuids: 4 hw.vmm.topology.cpuid_leaf_b: 1 hw.vmm.topology.cores_per_package: 1 hw.vmm.topology.threads_per_core: 1 hw.vmm.create: beavis hw.vmm.destroy: beavis hw.vmm.force_iommu: 0 hw.vmm.trace_guest_exceptions: 0 hw.vmm.ipinum: 251 hw.vmm.halt_detection: 1 > - host filesystem underlying hdd1 and hdd2 It's ZFS. Thank you! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On 03/06/2015 02:04, John-Mark Gurney wrote: > Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300: >> I guess you are running bhyve through the shell script vmrun.sh? >> I am doing everything by hand. > > Correct: > sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0 > > It's nice.. shutdown -r now and shutdown -p now both work exactly as > you'd expect them to... :) > Yes, it's quite convenient. The only drawback is that currently it can't pass multiple -d options to bhyveload and I need that from time to time. Probably not hard to change that. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
