Re: bhyve - windows7 installation
On 02/03/2018 21:01, Ludovit Koren wrote: > I applied the patch and I got the following error for usb in windows: > > This device cannot find enough free resources that it can use. (Code 12) > > If you want to use this device, you will need to disable one of the other > devices on this system. > > > I don't know how to proceed The hack that I shared is just a hack around some reasonable restrictions in bhyve. I don't know anything about Windows internals, so can't help further. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve - windows7 installation
On 02/03/2018 17:11, Rodney W. Grimes wrote: > Andriy, > Thanks for providing this patch, I tried to find the repository > it is in with google and did not, is this a private repository? Yes, a repo for my private development. > Either way would it be ok with you if I published, with attribution > of cource, your patch first from my people.freebsd.org/~rgrimes#bhyve > page, and possibly later from a wiki page? I'll properly dress it > in "this has security implications" and enumerate what those are. Yes, I am completely okay with it. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve - windows7 installation
On 01/03/2018 16:06, Rodney W. Grimes wrote:
> Due to the design of the IOMMU you can only manage IO space in page
> (4096 on x86) granually sizes. The device your trying to pass in
> has a 1024 byte memory region that is part of a 4096 byte page that
> may have other things in it.
>
> At this time bhyve does not have any way to deal with this, though some
> other hypervisors have techniques that make this work.
>
> I do not have or know of any list of USB controller cards that
> have 4k aligned and 4k sized BAR's.
I have this local hack for that problem.
It comes without any warranty and its use is completely at your own risk.
commit 74e0a8d1ae01c7aaabd7d965958b735c7cf18871
Author: Andriy Gapon <a...@icyb.net.ua>
Date: Fri Nov 17 20:17:57 2017 +0200
bhyve: allow BAR sizes that are not page aligned by rounding them up
This is based on the assumption that drivers won't access the added space.
diff --git a/usr.sbin/bhyve/pci_passthru.c b/usr.sbin/bhyve/pci_passthru.c
index f314679d912b0..14c1384c6c8f8 100644
--- a/usr.sbin/bhyve/pci_passthru.c
+++ b/usr.sbin/bhyve/pci_passthru.c
@@ -563,13 +563,20 @@ cfginitbar(struct vmctx *ctx, struct passthru_softc *sc)
size = bar.pbi_length;
if (bartype != PCIBAR_IO) {
- if (((base | size) & PAGE_MASK) != 0) {
+ if ((base & PAGE_MASK) != 0) {
warnx("passthru device %d/%d/%d BAR %d: "
- "base %#lx or size %#lx not page aligned\n",
+ "base %#lx not page aligned\n",
sc->psc_sel.pc_bus, sc->psc_sel.pc_dev,
- sc->psc_sel.pc_func, i, base, size);
+ sc->psc_sel.pc_func, i, base);
return (-1);
}
+ if ((size & PAGE_MASK) != 0) {
+ warnx("passthru device %d/%d/%d BAR %d: "
+ "size %#lx not page aligned\n",
+ sc->psc_sel.pc_bus, sc->psc_sel.pc_dev,
+ sc->psc_sel.pc_func, i, size);
+ size = round_page(size);
+ }
}
/* Cache information about the "real" BAR */
--
Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve passthrough problems
On 22/01/2018 16:54, Harry Schmalzbauer wrote: > Inderictly related, but also a big improvement for bhyve, could you have > a minute on that too: > https://lists.freebsd.org/pipermail/freebsd-virtualization/2017-November/005875.html What you request, has already been committed once and then reverted: https://svnweb.freebsd.org/base?view=revision=282922 https://svnweb.freebsd.org/base?view=revision=283168 Unfortunately, an 'upcoming commit' hasn't come yet. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve/amd: interrupt delivered when it shouldn't be?
On 30/12/2017 01:45, Peter Grehan wrote: > Hi Andriy, > >> The hardware is AMD. > > Ryzen ? No, a pretty old family 10h. >> But what I see suggests that at this point a Local APIC timer interrupt gets >> delivered to the thread. And that causes all the mess as the thread holding >> the >> spinlock gets preempted. >> >> Does this ring a bell to anyone? > > I have seen something similar to this after about ~20 mins when doing a > current > -j 16 buildworld in a guest, with the symptom being a spinlock timeout, with > one > vCPU spinning in > > smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x352/frame > 0xfe02c80098d0 > smp_masked_invlpg() at smp_masked_invlpg+0x4c/frame 0xfe02c8009900 > pmap_invalidate_page() at pmap_invalidate_page+0x191/frame 0xfe02c8009950 > pmap_ts_referenced() at pmap_ts_referenced+0x7b3/frame 0xfe02c8009a00 > vm_pageout() at vm_pageout+0xe04/frame 0xfe02c8009a70 > > ... and all the others eventually spinning on that held lock. This looks very similar to what I see as well. > However, NMIs are still able to get through (the post-panic ddb NMI IPI) so > the > VM isn't completely locked up - either an interrupt is missed, or a write > isn't > seen by the vCPU issuing the tlb shootdown. Yes, I can also enter ddb and remote kgdb. >> Is there any suspect code? > > Not sure yet, but the interrupt-injection path could do with a close > inspection. > >> It seems that we set v_intr_masking bit, so the rFLAGS / eFLAGS should be >> completely virtualized. So, maybe a hardware issue? > > Hard to say. Running with all vCPUs pinned makes the problem go away, but > that > could just mean the issue is isolated to when vCPUs migrate. I was going to try pinned CPUs and wired memory next, so your observation narrows the problem scope. Thank you! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
bhyve/amd: interrupt delivered when it shouldn't be?
First, about the setup. It's a FreeBSD/amd64 head guest on a FreeBSD/amd64 head host. The hardware is AMD. The hypervisor is bhyve. Under a certain specific load, that involves a lot of page faults and IPI-s, I see the guest system getting stuck. This is pretty consistent. Typically I find a thread spinning on smp_ipi_mtx. And an owner of the mutex appears to be in mi_switch() -> sched_switch(). The debugging data that I have is somewhat flaky, but it seems that the owner is typically in this code path: smp_targeted_tlb_shootdown -> ipi_send_cpu -> native_lapic_ipi_raw smp_targeted_tlb_shootdown holds smp_ipi_mtx. native_lapic_ipi_raw, in this setup, performs the following manipulations: saveintr = intr_disable(); ... intr_restore(saveintr); The interrupts are already disabled when this function is entered, because smp_ipi_mtx is a spinlock and our spinlock implementation disables interrupts. So, intr_restore() in this case should be a NOP (BTW, it's implemented via popf). But what I see suggests that at this point a Local APIC timer interrupt gets delivered to the thread. And that causes all the mess as the thread holding the spinlock gets preempted. Does this ring a bell to anyone? Is there any suspect code? It seems that we set v_intr_masking bit, so the rFLAGS / eFLAGS should be completely virtualized. So, maybe a hardware issue? Thank you! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
various fixes and improvements for AMD-Vi
I have created a fake GitHub pull request to try to get some review for a few AMD-specific changes to vmm. I would like to ask for a wider review of two of those changes. They require a bit of knowledge of the acpi and pci bus code (msi setup, etc): https://github.com/avg-I/freebsd/pull/1/commits/b4ce26a221fe2ab1e3ab1d373575131545b76aa6 https://github.com/avg-I/freebsd/pull/1/commits/eecf82121fc4fedc577af4b3c3b4de455ae6dad8 Thank you! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: problem with pass-through on amd
On 14/11/2017 06:22, Anish wrote: > [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh > irq256: ivhd0:fault 0 0 > irq257: ivhd1:fault 0 0 After I fixed the MSI setup problem that I described in the previous email I was finally able to see the real problem. Now I have: irq263: ivhd0:fault9 0 dev.ivhd.0.event_tail: 240 dev.ivhd.0.event_head: 240 dev.ivhd.0.event_intr_count: 9 I've got a bunch of messages like these in the log: [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x70f70400x30<PR,RW>] [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x2fdf0400x30<PR,RW>] (BTW, there seems to be a missing space before 0x30) Now it's obvious what the problem is. The controller has a RID of 0xa00 (its PCI locator is 10:0:0) but the requests are coming from 0xa01. The card actually has another PCI device at that location (10:0:1), it's a vestigial IDE controller (in a sense that it is not connected to any ports, so it cannot really provide any functionality). I've passed both of the PCI device to bhyve and everything started working. Thanks! I've googled a little bit and it seems that this is not an uncommon problem. Linux has a bunch of quirks for this and similar kinds of problems (they call it PCI or DMA aliasing): http://elixir.free-electrons.com/linux/v4.8.16/source/drivers/pci/quirks.c#L3701 The comments say that sometimes requests come from devices that do not exist at all (or hidden beyond non-transparent bridges). An interesting new world for me :-) By the way, my understanding is that the alias IVHD entries in IVRS are designed to report such aliasing issues when they are known to the platform. But it appears to me that FreeBSD currently ignores those aliases. Also, it looks like we are not making any use of IVHD entries beyond printing them. Based on the specification I think that we should use IVHD flags to set up appropriate bits in the corresponding device table entries. But we are not doing that. That's probably okay since we do not support the pass-though for devices with complex properties anyway. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: problem with pass-through on amd
On 14/11/2017 06:22, Anish wrote: > Also ivhd has fault interrupt enabled which is very helpful in debugging: > > [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh > irq256: ivhd0:fault 0 0 > irq257: ivhd1:fault 0 0 > Anish, I have made several interesting discoveries regarding my problem. One of them is that actually there were some IOMMU log events: dev.ivhd.0.event_tail: 240 dev.ivhd.0.event_head: 0 dev.ivhd.0.event_intr_count: 0 But there were no interrupts and the events are unconsumed and unreported. I examined MSI configuration of the IOMMU PCI device and the address and data registers were zeroed out. I looked at dmesg and at the code and I realized why that happened. So, first of all, I pre-load vmm via loader.conf. Probably as a result of that the ivhd device attaches before any bridges and buses on my system. And amdvi_alloc_intr_resources() does a rather untypical thing, it configures an MSI for a PCI device by directly writing to its configuration registers. The PCI bus code is completely unaware of those changes and it wipes them out in pci_add_child() -> pci_cfg_restore(). Also, I think that even if ivhd attached after the root PCI bus, then what it does would be still unsafe. I think that, for example, a suspend-resume cycle would wipe out the MSI configuration too. I think that in that case we should better use pci methods to configure MSI. Now, why does ivhd attach before the root Host-PCI bridge and what can we do to fix the order? ivrs_drv.c has this code: /* * Load this module at the end after PCI re-probing to configure interrupt. */ DRIVER_MODULE_ORDERED(ivhd, acpi, ivhd_driver, ivhd_devclass, 0, 0, SI_ORDER_ANY); But apparently this SI_ORDER_ANY does not help much. It affects only the driver registration order, but not the device probe and attachment order. This code is far more significant: ivhd_devs[i] = BUS_ADD_CHILD(parent, 1, "ivhd", i); ivhd passes 1 as the order. This is a very high order for the acpi bus. As a comment in acpi_probe_child() says: /* * Create a placeholder device for this node. Sort the * placeholder so that the probe/attach passes will run * breadth-first. Orders less than ACPI_DEV_BASE_ORDER * are reserved for special objects (i.e., system * resources). */ where ACPI_DEV_BASE_ORDER is 100. For example, order of the Host-PCI bridge on my system is 120. I must note that this is important only of vmm is preloaded (which is probably not an extremely rare case). If vmm is loaded after the system is booted then, of course, ivhd will be probed after the PCI buses / bridges. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: problem with pass-through on amd
On 14/11/2017 06:22, Anish wrote: > Hi Andriy, > Newer AMD IOMMU has EFR capability which is missing here, something like: > ivhd0: PCI cap 0x190b640f@0x40 feature:19<IOTLB,EFR> > > Is this an old chipset? Yes, this is an old 900 series chipset (990X). > Also ivhd has fault interrupt enabled which is very > helpful in debugging: > > [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh > irq256: ivhd0:fault 0 0 > irq257: ivhd1:fault 0 0 irq256: ivhd0:fault0 0 > Another thing to worth checking is PCI config space of ivhd/IOMMU if you have not already done so. For what should I look there? -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: problem with pass-through on amd
On 13/11/2017 22:41, Anish wrote: > Hi Andriy, > >>What's suspicious is that there are no interrupts either for the > ppt device on the host or the ahci device in the guest. > > I think most likely AMD IOMMU/amdvi is not enabled, by default it is disabled > and can be confirmed by sysctl. > > [root@ryzen /usr/home/anish]# sysctl hw.vmm.amdvi > hw.vmm.amdvi.domain_id: 3 > hw.vmm.amdvi.disable_io_fault: 0 > hw.vmm.amdvi.ptp_level: 4 > hw.vmm.amdvi.host_ptp: 1 > hw.vmm.amdvi.enable: 1 > hw.vmm.amdvi.count: 2 > > If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should work. I followed your how-to, so I didn't make this mistake :-) $ sysctl hw.vmm.amdvi hw.vmm.amdvi.domain_id: 3 hw.vmm.amdvi.disable_io_fault: 0 hw.vmm.amdvi.ptp_level: 4 hw.vmm.amdvi.host_ptp: 1 hw.vmm.amdvi.enable: 1 hw.vmm.amdvi.count: 1 Could it be that AMD-Vi support is broken in the hardware or BIOS? Are there any checks that I could do? Here is what I see in dmesg, but it is not a full dump of IVRS, of course: $ dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu' Table 'IVRS' at 0xbdafaf80 ACPI: IVRS 0xBDAFAF80 F8 (v01 AMDRD890S 00202031 AMD ) AMD-Vi IVRS VAsize = 64 PAsize = 52 GVAsize = 0 flags:0 ivhd0: on acpi0 ivhd0: Unknown dev entry:0xff ivhd0: Flag:3e<PassPW,ResPassPW,Isoc,IotlbSup,Coherent> ivhd0: max supported paging level:7 restricting to: 4 ivhd0: device supported range [0x0 - 0xb01] ivhd0: device [0xa0 - 0xa0]config:d7<INIT,ExtInt,NMI,LINT0> ivhd0: PCI cap 0x10b540f@0x40 feature:1 pci0: at device 0.2 (no driver attached) > On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon <a...@freebsd.org > <mailto:a...@freebsd.org>> wrote: > > > I have a FreeBSD guest on a FreeBSD host, both amd64 head. > I decided to experiment with PCI pass-through and used an AHCI controller > for > the experiment. It seems like that partially works. The guest detects > the > controller and attaches a driver to it, the messages (reported > capabilities, > etc) look identical to those on the host. But the guest can not detect > any > disks behind the controller. From what I can tell, the guest has no > problems > accessing PCI configuration registers and a memory mapped BAR. Not sure > about > the I/O ports. What's suspicious is that there are no interrupts either > for the > ppt device on the host or the ahci device in the guest. Maybe that is > what > causes the failure to see the disks. > > When the guest driver attaches to the controller I see these messages on > the > host: > ppt0: attempting to allocate 1 MSI vectors (1 supported) > ppt0: using IRQ 265 for MSI > and in the guest: > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > > But vmstat -i does not report any interrupts. > > Just in case, the controller has some complex topology where the AHCI PCI > device > is behind two PCI-PCI bridges from the main bus. But I think that that > should > not matter and the guest needs to see only the AHCI device and the host > handles > the bridges. > > Here are some more messages from the guest: > ahci0: port > 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem > 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS > ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports > ahcich0: at channel 0 on ahci0 > ahcich0: Caps: CPD ESP FBSCP > ahcich1: at channel 1 on ahci0 > ahcich1: Caps: CPD ESP FBSCP > ahcich0: AHCI reset... > ahcich0: SATA connect time=100us status=0123 > ahcich0: AHCI reset: device found > ahcich1: AHCI reset... > ahcich1: SATA connect time=100us status=0123 > ahcich1: AHCI reset: device found > ahcich1: AHCI reset: device ready after 100ms > ahcich1: Poll timeout on slot 1 port 15 > ahcich1: is cs 0002 ss rs 0002 tfd 150 serr > > cmd 00318017 > ahcich0: AHCI reset: device ready after 200ms > ahcich0: Poll timeout on slot 1 port 15 > ahcich0: is cs 0002 ss rs 0002 tfd 150 serr > > cmd 00318017 > And some more... > > Any ideas or suggestions? > Or maybe some additional information from me? > > Thank you! > > -- > Andriy Gapon > ___ > freebsd-virtualization@freebsd.org &g
problem with pass-through on amd
I have a FreeBSD guest on a FreeBSD host, both amd64 head. I decided to experiment with PCI pass-through and used an AHCI controller for the experiment. It seems like that partially works. The guest detects the controller and attaches a driver to it, the messages (reported capabilities, etc) look identical to those on the host. But the guest can not detect any disks behind the controller. From what I can tell, the guest has no problems accessing PCI configuration registers and a memory mapped BAR. Not sure about the I/O ports. What's suspicious is that there are no interrupts either for the ppt device on the host or the ahci device in the guest. Maybe that is what causes the failure to see the disks. When the guest driver attaches to the controller I see these messages on the host: ppt0: attempting to allocate 1 MSI vectors (1 supported) ppt0: using IRQ 265 for MSI and in the guest: ahci0: attempting to allocate 1 MSI vectors (1 supported) ahci0: using IRQ 279 for MSI But vmstat -i does not report any interrupts. Just in case, the controller has some complex topology where the AHCI PCI device is behind two PCI-PCI bridges from the main bus. But I think that that should not matter and the guest needs to see only the AHCI device and the host handles the bridges. Here are some more messages from the guest: ahci0: port 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 ahci0: attempting to allocate 1 MSI vectors (1 supported) ahci0: using IRQ 279 for MSI ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports ahcich0: at channel 0 on ahci0 ahcich0: Caps: CPD ESP FBSCP ahcich1: at channel 1 on ahci0 ahcich1: Caps: CPD ESP FBSCP ahcich0: AHCI reset... ahcich0: SATA connect time=100us status=0123 ahcich0: AHCI reset: device found ahcich1: AHCI reset... ahcich1: SATA connect time=100us status=0123 ahcich1: AHCI reset: device found ahcich1: AHCI reset: device ready after 100ms ahcich1: Poll timeout on slot 1 port 15 ahcich1: is cs 0002 ss rs 0002 tfd 150 serr cmd 00318017 ahcich0: AHCI reset: device ready after 200ms ahcich0: Poll timeout on slot 1 port 15 ahcich0: is cs 0002 ss rs 0002 tfd 150 serr cmd 00318017 And some more... Any ideas or suggestions? Or maybe some additional information from me? Thank you! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
dos in bhyve
I wonder if anyone was able to run any DOS variant in bhyve. I tried FreeDOS using BHYVE_UEFI_CSM firmware, but it seems to just hang. I also tried adding a frame buffer device, it seems that there is some reaction to key presses (RET) sent via VNC, but the screen stays black, so it's impossible to tell if there are any prompts and what happens. In any case, after a few RET-s the guest crashes with a triple fault. Seems like that happens when it enables paging. Some data from the VM state after the crash. Last instructions: mov%eax,%cr0 mov$0x11,%edi <=== triple fault here cr0[0] 0x8033 cr3[0] 0x00115000 cr4[0] 0x06a0 I wonder if it's the value of CR4 that causes the trouble (PAE bit set). -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
bhyve -g gets killed by EPIPE
I am trying to follow this guide: https://wiki.freebsd.org/bhyve/DebuggingWithGdb It seems to work, once or twice, but after that when I active kdb in the guest bhyve would terminate with status 141. I interpret it as signal 13, EPIPE. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
vtbd stuck i/o
90112,
bio_completed = 0,
bio_children = 0,
bio_inbed = 0,
bio_parent = 0xf80003b8b8d0,
bio_t0 = {
sec = 23228,
frac = 3190990625700496616
},
bio_task = 0x0,
bio_task_arg = 0x0,
bio_classifier1 = 0x0,
bio_classifier2 = 0x0,
bio_pblkno = 32562128
}
--
Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
bhyve: disable msi and msix on virtio reset?
A write of a zero to VTCFG_R_STATUS initiates a virtio device reset via vc_reset. Typically this means a call to vi_reset_dev() which resets a bunch of fields in virtio_softc, but does not touch a corresponding pci_devinst (hanging off vs_pi) at all. Among other things this means that PCI MSI and MSI-X states remain unchanged. One of the consequences is that we keep using virtio_config_size of 24 if MSI-X is enabled. Should the virtio status reset also reset the PCI state? One practical problem that I see is with illumos fast reboot where the illumos virtio driver assumes that the status reset is sufficient to return a device to a state like after a clean (full) reboot. Thank you. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: assertion in pci_vtblk_proc
On 15/03/2016 16:12, Peter Grehan wrote: > Hi Andriy, > >> While running some fs-related workloads in a Gentoo guest bhyve crashed with >> the >> following assertion: >> >> Assertion failed: (n >= 2 && n <= BLOCKIF_IOV_MAX + 2), function >> pci_vtblk_proc, >> file /usr/src/usr.sbin/bhyve/pci_virtio_block.c > ... >> Please let me know if you would like more details. > > Would you be able to get the value of 'n' from gdb ? > This is possibly a similar issue to that seen under Windows, which I had an > attempt at fixing with r282922 (subsequently reverted). In my case n is zero, so it's likely a different issue, I guess. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
bhyve vtblk error
The following happened with an illumos VM in bhyve. Usually I reboot that VM by first powering it off using illumos 'poweroff' command. But yesterday I tried to do that using 'reboot' command. Apparently in that case illumos performs a warm reboot where a new kernel instance is executed without the full hardware reset and firmware boot. So, I got the following error in the guest during the reboot: Block device: blkdev@0,0, blkdev0 blkdev0 is /pci@0,0/pci1af4,2@4/blkdev@0,0 /pci@0,0/pci1af4,2@4/blkdev@0,0 (blkdev0) online WARNING: blkdev0: Invalid media block size (0) WARNING: blkdev0: Invalid media block size (0) NOTICE: Cannot read the pool label from '/pci@0,0/pci1af4,2@4/blkdev@0,0:a' NOTICE: spa_import_rootpool: error 5 Cannot mount root on /pci@0,0/pci1af4,2@4/blkdev@0,0:a fstype zfs At the same time bhyve printed the following message: vtblk: read from CFGVEC: bad size 4 P.S. There is another problem that perhaps is worth some attention as well. The guest paniced after it failed to import the root pool. bhyve got stuck in a weird way after that: vm exit[2] vm exit[1] vm exit[3] reason SVM rip 0xfb8440aa inst_length 0 exitcode0x7f exitinfo1 0 exitinfo2 0 reason SVM rip 0xfb8440aa inst_length 0 exitcode0x7f exitinfo1 0 exitinfo2 0 reason SVM rip 0xfb8440aa inst_length 0 exitcode0x7f exitinfo1 0 exitinfo2 0 ^T load: 0.98 cmd: bhyve 87140 [suspended] 20861.13r 643.47u 41127.33s 0% 3727592k As you can see the process somehow became suspended and I was not able to recover it (SIGKILL didn't help) and had to reboot the host. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Illumos boot
On 15/10/2015 18:56, Peter Grehan wrote: > Hi Matt, > >>> -s 0,hostbridge > > Things should work if you leave out the hostbridge. The PCIe capability > that is tacked on to this will make Illumos use MSI/MSIx for the virtio > adapter which apparently hits a bug in the driver. Without it, the > virtio driver will fall back to legacy interrupts. This also means that > the virtio adapter will be confined to slots 3/4/5/6. > > [root@smartos ~]# echo ::interrupts | mdb -k > IRQ Vect IPL BusTrg Type CPU Share APIC/INT# ISR(s) > 10x40 5 ISAEdg Fixed 0 1 0x0/0x1 i8042_intr > 30xb1 12 ISAEdg Fixed 0 1 0x0/0x3 asyintr > 40xb0 12 ISAEdg Fixed 0 1 0x0/0x4 asyintr > 90x81 9 PCILvl Fixed 1 1 0x0/0x9 acpi_wrapper_isr > 12 0x41 5 ISAEdg Fixed 1 1 0x0/0xc i8042_intr > 16 0x42 5 PCILvl Fixed 1 1 0x0/0x10 ahci_intr > 17 0x43 5 PCILvl Fixed 0 1 0x0/0x11 ahci_intr > 18 0x60 6 PCILvl Fixed 1 1 0x0/0x12 virtio_intx_dispatch > 160 0xa0 0 Edg IPIall 0 - poke_cpu > 208 0xd0 14 Edg IPIall 1 - kcpc_hw_overflow_intr > 209 0xd1 14 Edg IPIall 1 - cbe_fire > 210 0xd3 14 Edg IPIall 1 - cbe_fire > 240 0xe0 15 Edg IPIall 1 - xc_serv > 241 0xe1 15 Edg IPIall 1 - apic_error_intr > > > (I believe Andriy Gapon (cc'd) has a fix for this in Illumos) Yes, it's here https://www.illumos.org/rb/r/86/ I plan to RTI that change soon-ish. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: centos 7.1 with multiple virtual processors
On 15/07/2015 10:02, Neel Natu wrote: > > Could you update the host with the following patch? > https://people.freebsd.org/~neel/patches/ktr_stray_nmi.patch > > It enables KTR_GEN logging by default and turns it off when a VM-exit > due to NMI is detected. If there is a pattern to the NMIs then it can > be correlated with the guest activity around that time. Neel, sorry that it took so long to gather the additional data. Yesterday I got another host NMI while running a VM with KTR turned on. I panic-ed the system from the debugger prompt after the NMI. Below is several dozen of last messages taken from the crash dump. It does not look like there was a guest NMI or, at least, it didn't get a chance to be logged. Please let me know if the full log dump would be more useful. Initially, the last reported address 0x7f8c1185129b looked suspicious to me, but grepping the dump for 'extintr' I see many addresses like that. $ ktrdump -N /boot/kernel.dn/kernel -M /var/crash/vmcore.21 | head -40 index trace -- - 253313 vm centos7[0]: Resume execution at 0x7f8c1185129b 253312 vm centos7[0]: vmcb clean 0x3bf 253311 vm centos7[0]: handled extintr vmexit at 0x7f8c1185129b/0 253310 vm centos7[0]: Resume execution at 0x8105059c 253309 vm centos7[0]: vmcb clean 0x3bf 253308 vm centos7[0]: Guest interrupt blocking cleared due to rip change: 0x81050596/0x8105059c 253307 vm centos7[0]: vcpu state changed from frozen to running 253306 vm centos7[0]: nextrip updated to 0x8105059c after instruction decoding 253305 vm centos7[0]: inst_emul fault accessing gpa 0xfed000f0 253304 vm centos7[0]: vcpu state changed from running to frozen 253303 vm centos7[0]: unhandled nptfault vmexit at 0x81050596/0 253302 vm centos7[0]: inst_emul fault for gpa 0xfed000f0/0x10004 at rip 0x81050596 253301 vm centos7[0]: Resume execution at 0x8105058f 253300 vm centos7[0]: vmcb clean 0x3bf 253299 vm centos7[0]: Guest interrupt blocking cleared due to rip change: 0x8105058d/0x8105058f 253298 vm centos7[0]: vcpu state changed from frozen to running 253297 vm centos7[0]: nextrip updated to 0x8105058f after instruction decoding 253296 vm centos7[0]: inst_emul fault accessing gpa 0xfed00108 253295 vm centos7[0]: vcpu state changed from running to frozen 253294 vm centos7[0]: unhandled nptfault vmexit at 0x8105058d/0 253293 vm centos7[0]: inst_emul fault for gpa 0xfed00108/0x10006 at rip 0x8105058d 253292 vm centos7[0]: Resume execution at 0x8105057c 253291 vm centos7[0]: vmcb clean 0x3bf 253290 vm centos7[0]: Guest interrupt blocking cleared due to rip change: 0x81050576/0x8105057c 253289 vm centos7[0]: vcpu state changed from frozen to running 253288 vm centos7[0]: nextrip updated to 0x8105057c after instruction decoding 253287 vm centos7[0]: inst_emul fault accessing gpa 0xfed000f0 253286 vm centos7[0]: vcpu state changed from running to frozen 253285 vm centos7[0]: unhandled nptfault vmexit at 0x81050576/0 253284 vm centos7[0]: inst_emul fault for gpa 0xfed000f0/0x10004 at rip 0x81050576 253283 vm centos7[0]: Resume execution at 0x8104accc 253282 vm centos7[0]: vmcb clean 0x3b7 253281 vm centos7[0]: Clearing V_IRQ interrupt injection 253280 vm centos7[0]: Guest interrupt blocking cleared due to rip change: 0x8104acc6/0x8104accc 253279 vm centos7[0]: vcpu state changed from frozen to running 253278 vm centos7[0]: vlapic_update_ppr 0x00 253277 vm centos7[0]: vlapic_process_eoi isr7 0x 253276 vm centos7[0]: vlapic_process_eoi isr6 0x And a snippet from the stack-trace: ... #9 0x8081ed84 in trap (frame=0xfe01dc531f30) at /usr/src/sys/amd64/amd64/trap.c:372 #10 0x808074e3 in nmi_calltrap () at /usr/src/sys/libkern/explicit_bzero.c:28 #11 0x81c92327 in enable_gintr () at /usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895 #12 0x81c91c57 in svm_vmrun (arg=0xfe006d29c000, vcpu=0, rip=-2130377316, pmap=0xf801e4771138, evinfo=0xfe02b8c45820) at /usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:2022 #13 0x81c7661e in vm_run (vm=0xfe0076277000, vmrun=0xf8013fe43000) at /usr/src/sys/modules/vmm/../../amd64/vmm/vmm.c:1643 #14 0x81c79b78 in vmmdev_ioctl (cdev=, cmd=, data=0xf8013fe43000 "", fflag=-1, td=0x0) at /usr/src/sys/modules/vmm/../../amd64/vmm/vmm_dev.c:392 ... -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: grub-bhyve: editing boot commands
On 30/07/2015 21:18, Peter Grehan wrote: Hi Andriy, In the grub-bhyve menu of boot entries I can press 'e' and enter a screen where I can modify boot commands for that entry. The screen has the following help information at the bootom: Minimum Emacs-like screen editing is supported. TAB lists completions. Press Ctrl-x or F10 to boot, Ctrl-c or F2 for a command-line or ESC to discard edits and return to the GRUB menu. I can make any edits in that screen, but of the mentioned key presses only TAB and ESC work as advertised. Ctrl-x, F10, Ctrl-c, F2 are all ignored. Is there a way to make them work? Fixed upstream https://github.com/grehan-freebsd/grub2-bhyve/commit/a2265476e97afb9b67cfca0d1d9e5be8def01f33 Thank you very much! Do you plan to release a new version and update the port soon? :) -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
bhyve vs signals
I couldn't find any information in bhyve(8) on how bhyve responds to various signals. It seems that at least SIGTERM is intercepted and translated to an ACPI power event. How about other signals? What's the best signal to use to kill a hung VM that can not be stopped gracefully? -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: centos 7.1 with multiple virtual processors
On 23/06/2015 05:37, Neel Natu wrote:
Hi Andriy,
FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus
fine on my host with 8 physical cores.
I have some questions about your setup inline.
On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon a...@freebsd.org wrote:
If I run a CentOS 7.1 VM with more than one CPU more often than not it would
hang on startup and bhyve would start spinning.
The following are the last messages seen in the VM:
Switching to clocksource hpet
[ cut here ]
WARNING: at kernel/time/clockevents.c:239
clockevents_program_event+0xdb/0xf0()
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1
Hardware name: BHYVE, BIOS 1.00 03/14/2014
cab5bdb6 88003fc03e08 81604eaa
88003fc03e40 8106e34b 800f423f 800f423f
81915440 88003fc03e50
Call Trace:
IRQ [81604eaa] dump_stack+0x19/0x1b
[8106e34b] warn_slowpath_common+0x6b/0xb0
[8106e49a] warn_slowpath_null+0x1a/0x20
[810ce6eb] clockevents_program_event+0xdb/0xf0
[810cf211] tick_handle_periodic_broadcast+0x41/0x50
[81016525] timer_interrupt+0x15/0x20
[8110b5ee] handle_irq_event_percpu+0x3e/0x1e0
[8110b7cd] handle_irq_event+0x3d/0x60
[8110e467] handle_edge_irq+0x77/0x130
[81015cff] handle_irq+0xbf/0x150
[81077df7] ? irq_enter+0x17/0xa0
[816172af] do_IRQ+0x4f/0xf0
[8160c4ad] common_interrupt+0x6d/0x6d
EOI [8126e359] ? selinux_inode_alloc_security+0x59/0xa0
[811de58f] ? __d_instantiate+0xbf/0x100
[811de56f] ? __d_instantiate+0x9f/0x100
[811de60d] d_instantiate+0x3d/0x70
[8124d748] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130
[8124da82] __create_file+0x1c2/0x2c0
[81a6c6bf] ? set_graph_function+0x1f/0x1f
[8124dbcb] debugfs_create_dir+0x1b/0x20
[8112c1ce] tracing_init_dentry_tr+0x7e/0x90
[8112c250] tracing_init_dentry+0x10/0x20
[81a6c6d2] ftrace_init_debugfs+0x13/0x1fd
[81a6c6bf] ? set_graph_function+0x1f/0x1f
[810020e8] do_one_initcall+0xb8/0x230
[81a45203] kernel_init_freeable+0x18b/0x22a
[81a449db] ? initcall_blacklist+0xb0/0xb0
[815f33f0] ? rest_init+0x80/0x80
[815f33fe] kernel_init+0xe/0xf0
[81614d3c] ret_from_fork+0x7c/0xb0
[815f33f0] ? rest_init+0x80/0x80
---[ end trace d5caa1cab8e7e98d ]---
A few questions to narrow this down:
- Is the host very busy when the VM is started (or what is the host
doing when this happened)?
The host typically is not heavily loaded. There is X server running and some
applications. I'd imagine that those could cause some additional latency but
not CPU starvation.
- How many vcpus are you giving to the VM?
- How many cores on the host?
I tried only 2 / 2.
At the same time sometimes there is one or more of spurious NMIs on the
_host_
system:
NMI ISA c, EISA ff
NMI ... going to debugger
Hmm, that's interesting. Are you using hwpmc to do instruction sampling?
hwpmc driver is in the kernel, but it was not used.
bhyve seems to spin here:
vmm.ko`svm_vmrun+0x894
vmm.ko`vm_run+0xbb7
vmm.ko`vmmdev_ioctl+0x5a4
kernel`devfs_ioctl_f+0x13b
kernel`kern_ioctl+0x1e1
kernel`sys_ioctl+0x16a
kernel`amd64_syscall+0x3ca
kernel`0x8088997b
(kgdb) list *svm_vmrun+0x894
0x813c9194 is in svm_vmrun
(/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895).
1890
1891static __inline void
1892enable_gintr(void)
1893{
1894
1895__asm __volatile(stgi);
1896}
1897
1898/*
1899 * Start vcpu with specified RIP.
Yeah, that's not surprising because host interrupts are blocked when
the cpu is executing in guest context. The 'enable_gintr()' enables
interrupts so it gets blamed by the interrupt-based sampling.
In this case it just means that the cpu was in guest context when a
host-interrupt fired.
I see. FWIW, that was captured with DTrace.
--
Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: centos 7.1 with multiple virtual processors
On 23/06/2015 10:26, Neel Natu wrote: Hi Andriy, On Mon, Jun 22, 2015 at 11:45 PM, Andriy Gapon a...@freebsd.org wrote: On 23/06/2015 05:37, Neel Natu wrote: Hi Andriy, FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus fine on my host with 8 physical cores. I have some questions about your setup inline. On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon a...@freebsd.org wrote: If I run a CentOS 7.1 VM with more than one CPU more often than not it would hang on startup and bhyve would start spinning. The following are the last messages seen in the VM: Switching to clocksource hpet [ cut here ] WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0() Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1 Hardware name: BHYVE, BIOS 1.00 03/14/2014 cab5bdb6 88003fc03e08 81604eaa 88003fc03e40 8106e34b 800f423f 800f423f 81915440 88003fc03e50 Call Trace: IRQ [81604eaa] dump_stack+0x19/0x1b [8106e34b] warn_slowpath_common+0x6b/0xb0 [8106e49a] warn_slowpath_null+0x1a/0x20 [810ce6eb] clockevents_program_event+0xdb/0xf0 [810cf211] tick_handle_periodic_broadcast+0x41/0x50 [81016525] timer_interrupt+0x15/0x20 [8110b5ee] handle_irq_event_percpu+0x3e/0x1e0 [8110b7cd] handle_irq_event+0x3d/0x60 [8110e467] handle_edge_irq+0x77/0x130 [81015cff] handle_irq+0xbf/0x150 [81077df7] ? irq_enter+0x17/0xa0 [816172af] do_IRQ+0x4f/0xf0 [8160c4ad] common_interrupt+0x6d/0x6d EOI [8126e359] ? selinux_inode_alloc_security+0x59/0xa0 [811de58f] ? __d_instantiate+0xbf/0x100 [811de56f] ? __d_instantiate+0x9f/0x100 [811de60d] d_instantiate+0x3d/0x70 [8124d748] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130 [8124da82] __create_file+0x1c2/0x2c0 [81a6c6bf] ? set_graph_function+0x1f/0x1f [8124dbcb] debugfs_create_dir+0x1b/0x20 [8112c1ce] tracing_init_dentry_tr+0x7e/0x90 [8112c250] tracing_init_dentry+0x10/0x20 [81a6c6d2] ftrace_init_debugfs+0x13/0x1fd [81a6c6bf] ? set_graph_function+0x1f/0x1f [810020e8] do_one_initcall+0xb8/0x230 [81a45203] kernel_init_freeable+0x18b/0x22a [81a449db] ? initcall_blacklist+0xb0/0xb0 [815f33f0] ? rest_init+0x80/0x80 [815f33fe] kernel_init+0xe/0xf0 [81614d3c] ret_from_fork+0x7c/0xb0 [815f33f0] ? rest_init+0x80/0x80 ---[ end trace d5caa1cab8e7e98d ]--- A few questions to narrow this down: - Is the host very busy when the VM is started (or what is the host doing when this happened)? The host typically is not heavily loaded. There is X server running and some applications. I'd imagine that those could cause some additional latency but not CPU starvation. Yup, I agree. Does this ever happen with a single vcpu guest? Never seen the problem with a single CPU so far. Also, never had that problem with FreeBSD guests. The other mystery is the NMIs the host is receiving. I (re)verified to make sure that bhyve/vmm.ko do not assert NMIs so it has to be something else on the host that's doing it ... But the correlation with the multi-CPU non-FreeBSD guests seems to be significant. P.S. meanwhile I found this old-ish thread that seems to describe exactly the problem I am seeing but on real hardware: http://thread.gmane.org/gmane.linux.kernel/1483297 -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: dtrace fbt vs vmm
On 22/06/2015 19:08, Peter Grehan wrote: Hi Andriy, I haven't investigated this issue further, but if there are any unsafe functions in vmm it would be nice to blacklist them in the fbt code. Perhaps blacklisting everything from vmm module could be a stopgap solution. Intel or AMD ? We've been able to run dtrace on Intel; less (to no) coverage on AMD so there's quite possibly bugs there. Peter, it's AMD again. Thanks! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: centos 7.1 with multiple virtual processors
On 22/06/2015 17:41, Peter Grehan wrote: Hi Andriy, If I run a CentOS 7.1 VM with more than one CPU more often than not it would hang on startup and bhyve would start spinning. Looks like an AMD host - what's the version of FreeBSD you are running there ? Yes, this is an AMD host, the version is head@283188. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
bhyve: centos 7.1 with multiple virtual processors
If I run a CentOS 7.1 VM with more than one CPU more often than not it would
hang on startup and bhyve would start spinning.
The following are the last messages seen in the VM:
Switching to clocksource hpet
[ cut here ]
WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0()
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1
Hardware name: BHYVE, BIOS 1.00 03/14/2014
cab5bdb6 88003fc03e08 81604eaa
88003fc03e40 8106e34b 800f423f 800f423f
81915440 88003fc03e50
Call Trace:
IRQ [81604eaa] dump_stack+0x19/0x1b
[8106e34b] warn_slowpath_common+0x6b/0xb0
[8106e49a] warn_slowpath_null+0x1a/0x20
[810ce6eb] clockevents_program_event+0xdb/0xf0
[810cf211] tick_handle_periodic_broadcast+0x41/0x50
[81016525] timer_interrupt+0x15/0x20
[8110b5ee] handle_irq_event_percpu+0x3e/0x1e0
[8110b7cd] handle_irq_event+0x3d/0x60
[8110e467] handle_edge_irq+0x77/0x130
[81015cff] handle_irq+0xbf/0x150
[81077df7] ? irq_enter+0x17/0xa0
[816172af] do_IRQ+0x4f/0xf0
[8160c4ad] common_interrupt+0x6d/0x6d
EOI [8126e359] ? selinux_inode_alloc_security+0x59/0xa0
[811de58f] ? __d_instantiate+0xbf/0x100
[811de56f] ? __d_instantiate+0x9f/0x100
[811de60d] d_instantiate+0x3d/0x70
[8124d748] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130
[8124da82] __create_file+0x1c2/0x2c0
[81a6c6bf] ? set_graph_function+0x1f/0x1f
[8124dbcb] debugfs_create_dir+0x1b/0x20
[8112c1ce] tracing_init_dentry_tr+0x7e/0x90
[8112c250] tracing_init_dentry+0x10/0x20
[81a6c6d2] ftrace_init_debugfs+0x13/0x1fd
[81a6c6bf] ? set_graph_function+0x1f/0x1f
[810020e8] do_one_initcall+0xb8/0x230
[81a45203] kernel_init_freeable+0x18b/0x22a
[81a449db] ? initcall_blacklist+0xb0/0xb0
[815f33f0] ? rest_init+0x80/0x80
[815f33fe] kernel_init+0xe/0xf0
[81614d3c] ret_from_fork+0x7c/0xb0
[815f33f0] ? rest_init+0x80/0x80
---[ end trace d5caa1cab8e7e98d ]---
At the same time sometimes there is one or more of spurious NMIs on the _host_
system:
NMI ISA c, EISA ff
NMI ... going to debugger
bhyve seems to spin here:
vmm.ko`svm_vmrun+0x894
vmm.ko`vm_run+0xbb7
vmm.ko`vmmdev_ioctl+0x5a4
kernel`devfs_ioctl_f+0x13b
kernel`kern_ioctl+0x1e1
kernel`sys_ioctl+0x16a
kernel`amd64_syscall+0x3ca
kernel`0x8088997b
(kgdb) list *svm_vmrun+0x894
0x813c9194 is in svm_vmrun
(/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895).
1890
1891static __inline void
1892enable_gintr(void)
1893{
1894
1895__asm __volatile(stgi);
1896}
1897
1898/*
1899 * Start vcpu with specified RIP.
--
Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
header type for bhyve host-pci bridge
diff --git a/usr.sbin/bhyve/pci_hostbridge.c b/usr.sbin/bhyve/pci_hostbridge.c index 54a25ae..5c9ea28 100644 --- a/usr.sbin/bhyve/pci_hostbridge.c +++ b/usr.sbin/bhyve/pci_hostbridge.c @@ -38,7 +38,7 @@ pci_hostbridge_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts) /* config space */ pci_set_cfgdata16(pi, PCIR_VENDOR, 0x1275); /* NetApp */ pci_set_cfgdata16(pi, PCIR_DEVICE, 0x1275); /* NetApp */ - pci_set_cfgdata8(pi, PCIR_HDRTYPE, PCIM_HDRTYPE_BRIDGE); + pci_set_cfgdata8(pi, PCIR_HDRTYPE, PCIM_HDRTYPE_NORMAL); pci_set_cfgdata8(pi, PCIR_CLASS, PCIC_BRIDGE); pci_set_cfgdata8(pi, PCIR_SUBCLASS, PCIS_BRIDGE_HOST); It seems that the normal header type is expected for a Host-PCI bridge. I see that on real hardware. Also, Linux complains about the current header type: pci :00:00.0: ignoring class 0x06 (doesn't match header type 01) pci :00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci :00:00.0: can't allocate child bus 00 from [bus 00] Same with lspci: $ lspci -v 00:00.0 Non-VGA unclassified device: Network Appliance Corporation Device 1275 !!! Invalid class for header type 01 Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=00, subordinate=00, sec-latency=0 I/O behind bridge: -0fff Memory behind bridge: -000f Prefetchable memory behind bridge: -000f Capabilities: [40] Express Root Port (Slot-), MSI 00 I think that PCIM_HDRTYPE_BRIDGE is for PCI-PCI bridges. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
bhyve - grub-bhyve without intermediate destroy
This is what happens if I create and run a VM by using grub-bhyve and bhyve, then exit from the VM via shutdown -r within it, and then try to run it again by using grub-bhyve and bhyve: Assertion failed: (error == 0), function fbsdrun_addcpu, file /usr/src/usr.sbin/bhyve/bhyverun.c, line 261 grub-bhyve apparently succeeds, but bhyve can't start up. Both invocation of bhyve are with -c 1. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve - grub-bhyve without intermediate destroy
On 22/06/2015 22:08, Peter Grehan wrote: Hi Andriy, This is what happens if I create and run a VM by using grub-bhyve and bhyve, then exit from the VM via shutdown -r within it, and then try to run it again by using grub-bhyve and bhyve: Assertion failed: (error == 0), function fbsdrun_addcpu, file /usr/src/usr.sbin/bhyve/bhyverun.c, line 261 grub-bhyve apparently succeeds, but bhyve can't start up. Both invocation of bhyve are with -c 1. This was fixed in grub2-bhyve upstream with https://github.com/grehan-freebsd/grub2-bhyve/commit/370fa455d41212282bf63cea7b048e87a821a31a I'm gathering a few more fixes before doing a point release of grub2-bhyve, at which point the FreeBSD port will be updated. You'll need to do a 'bhyvectl --destroy' until then, or build grub2-bhyve from upstream. Thank you! Will the release that you are planning include the ext4 64-bit patch? -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: corrupting zfs pools?
On 10/06/2015 09:24, Neel Natu wrote: I tried to repro on a Sempron 3850 APU but still no luck. Since you can repro it readily I'll follow up with you so we can narrow this down on your end. After narrowing down the problem by following suggestions from Neel it became obvious that the problem was caused by my local modifications to ZFS on the host system. Sorry for the noise and thank you very much for the help. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: corrupting zfs pools?
On 04/06/2015 07:37, Neel Natu wrote: Ok, there are some differences in our systems. The interesting ones are number of ASIDs (64 versus 65536), flush-by-asid capability, vmcb-clean capability and the number of cores. I was able to mimic all of these on my Opteron but still wasn't able to reproduce the issue. I am going to get a Sempron tomorrow which belongs to the same processor family as the Athlon II so hoping that it is easier to repro. BTW does this happen consistently on your system? Yes, it happened every time I tried that scenario. Thank you very much! If there is anything else that I can do on my side to help the investigation just let me know. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On 03/06/2015 02:04, John-Mark Gurney wrote: Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300: I guess you are running bhyve through the shell script vmrun.sh? I am doing everything by hand. Correct: sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0 It's nice.. shutdown -r now and shutdown -p now both work exactly as you'd expect them to... :) Yes, it's quite convenient. The only drawback is that currently it can't pass multiple -d options to bhyveload and I need that from time to time. Probably not hard to change that. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On 03/06/2015 18:53, John-Mark Gurney wrote:
Andriy Gapon wrote this message on Wed, Jun 03, 2015 at 10:10 +0300:
On 03/06/2015 02:04, John-Mark Gurney wrote:
Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300:
I guess you are running bhyve through the shell script vmrun.sh?
I am doing everything by hand.
Correct:
sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0
It's nice.. shutdown -r now and shutdown -p now both work exactly as
you'd expect them to... :)
Yes, it's quite convenient. The only drawback is that currently it can't
pass
multiple -d options to bhyveload and I need that from time to time.
Hmm... Looking at the script, it looks like you can (at least on HEAD):
d)
eval disk_dev${disk_total}=\${OPTARG}\
disk_total=$(($disk_total + 1))
This was only half of the equation, please see this:
https://reviews.freebsd.org/D2723
Probably not hard to change that.
It'd also be nice to allow you to switch the default from virtio-blk
to ahci when you're image doesn't support it..
Hmm.. It wouldn't be hard to try to use environment vars for the defaults
either...
Maybe it's time to move vmrun.sh to bhyverun, install it and give
it a proper man page?
JIMO, bhyveload(8) could be just merged into bhyve(8) and the latter should
behave like vmrun.sh does. bhyve(8) should retain an option to run a preloaded
kernel, so that things like bhyve-grub keep working.
--
Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
bhyve: corrupting zfs pools?
I am doing a simple experiment. I get FreeBSD image from here: ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz Then I run in bhyve with two additional disks created with truncate -s 4g: $ bhyveload -m 1G -d ~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test $ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img -s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s 3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l com2,/dev/nmdm0A -c 2 -m 1g test Note sectorsize=512/4096 options. Not sure if it's them that cause the trouble. Then, in the VM: $ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2 $ zfs create -p l2arc-test/ROOT/initial $ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f - Afterwards, zpool status -v reports no problem. But then I run zpool scrub and get the following in the end: $ zpool status -v pool: l2arc-test state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun 2 13:58:17 2015 config: NAMESTATE READ WRITE CKSUM l2arc-test ONLINE 0 0 9 mirror-0 ONLINE 0 018 vtbd1 ONLINE 0 025 vtbd2 ONLINE 0 023 errors: Permanent errors have been detected in the following files: /l2arc-test/ROOT/initial/usr/bin/svnlitesync /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz /l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a The same issue is reproducible with ahci-hd. My host system is a recent amd64 CURRENT as well. The hardware platform is AMD. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
bhyve: bhyveload, bhyve, bhyvectl --destroy
I am very new to bhyve, so sorry if I am asking something silly or obvious. I am using bhyve to speed up my testing and it seems that each time I need to restart a VM I need to go through the cycle of destroying it with bhyvectl --destroy, then re-loading a kernel with bhyveload and then actually booting the VM with bhyve. It seems that I have to do this even if I don't change th kernel between reboots. My first naive impression was that the point of bhyveload was to load the kernel once. Seems it ain't so? -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On 02/06/2015 17:51, Peter Grehan wrote: Hi Andriy, I am very new to bhyve, so sorry if I am asking something silly or obvious. I am using bhyve to speed up my testing and it seems that each time I need to restart a VM I need to go through the cycle of destroying it with bhyvectl --destroy, then re-loading a kernel with bhyveload and then actually booting the VM with bhyve. It seems that I have to do this even if I don't change th kernel between reboots. My first naive impression was that the point of bhyveload was to load the kernel once. Seems it ain't so? bhyveload does the job of what BIOS/boot0/1/2/loader would do on real h/w, so it has to be executed each time on restart. One optimization to the cycle you mentioned is that bhyvectl --destroy only has to be done when the VM is no longer needed i.e you can loop with bhyveload/bhyve. I see now. Thank you very much! BTW, and probably you are already aware of this, the documentation could use some work :-) I noticed at least the following: - bhyvectl is not documented - bhyve(8) and bhyveload(8) refer to vmm(4), which does not exist - bhyveload(8) does not mention that -d can be used multiple times - bhyve(8) has at least one incorrect reference to bhyveload(*4*) - bhyve(8) mentions vmnet - does that exist? But these are very minor things. bhyve rules :) -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: corrupting zfs pools?
On 02/06/2015 14:14, Andriy Gapon wrote: I am doing a simple experiment. I get FreeBSD image from here: ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz Then I run in bhyve with two additional disks created with truncate -s 4g: $ bhyveload -m 1G -d ~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test $ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img -s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s 3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l com2,/dev/nmdm0A -c 2 -m 1g test Note sectorsize=512/4096 options. Not sure if it's them that cause the trouble. Then, in the VM: $ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2 $ zfs create -p l2arc-test/ROOT/initial $ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f - Afterwards, zpool status -v reports no problem. But then I run zpool scrub and get the following in the end: $ zpool status -v pool: l2arc-test state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun 2 13:58:17 2015 config: NAMESTATE READ WRITE CKSUM l2arc-test ONLINE 0 0 9 mirror-0 ONLINE 0 018 vtbd1 ONLINE 0 025 vtbd2 ONLINE 0 023 errors: Permanent errors have been detected in the following files: /l2arc-test/ROOT/initial/usr/bin/svnlitesync /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz /l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a The same issue is reproducible with ahci-hd. My host system is a recent amd64 CURRENT as well. The hardware platform is AMD. I used the following monstrous command line to reproduce the test in qemu: $ qemu-system-x86_64 -smp 2 -m 1024 -drive file=/tmp/livecd2/R2.img,format=raw,if=none,id=bootd -device virtio-blk-pci,drive=bootd -drive file=/tmp/l2arc-test/hdd1,if=none,id=hdd1,format=raw -device virtio-blk-pci,drive=hdd1,logical_block_size=4096 -drive file=/tmp/l2arc-test/hdd2,id=hdd2,if=none,format=raw -device virtio-blk-pci,drive=hdd2,logical_block_size=4096 -drive file=/tmp/l2arc-test/ssd,id=ssd,if=none,format=raw -device virtio-blk-pci,drive=ssd,logical_block_size=4096 ... And several other variations of logical_block_size and physical_block_size. The tests a re very slow, but there are no checksum errors. So, I suspect guest memory corruption caused by bhyve. Perhaps the problem is indeed specific to AMD-V. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
bhyve: svm (amd-v) update
It seems that the bhyve_svm branch is a bit behind the latest interface changes in head. Is anyone working on sync-ing up the branch with the head? Some examples: - change of init method in vmm_ops - addition of resume, vlapic_init and vlapic_cleanup methods to vmm_ops - replacement of lapic_pending_intr and lapic_intr_accepted with vlapic_pending_intr and vlapic_intr_accepted - changes in struct vm_exit - VMEXIT_EPT_FAULT == VMEXIT_NESTED_FAULT Thanks! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
